Transcript
Chao: I want to introduce Pkl, and introduce it in high level terms. I want to spend most of the time in the editor and showing you what it’s like to use Pkl.
I’m one of the core maintainers of Pkl at Apple. It’s a pretty small team. There’s three of us maintaining Pkl, but it’s a great group. I think we’ve done some pretty awesome things throughout the years. I am an ex-musician. I went into tech by meandering through the musician lifestyle and getting a degree in music performance, and then suddenly realizing, I really like programming.
Then I got really obsessed with it and been doing it ever since. As far as tech, I’ve been all over the place. I’ve spent a bunch of my time doing services. I’ve also done apps. I’ve done iOS and Android. I’ve been a DevOps engineer. I’ve worn every single hat that you can wear, including being a designer for some reason. One of the things that I’ve done in the past that was also language related is I used to design children-oriented programming languages, which is not object-oriented. It’s not functional. It’s this stuff. It’s something meant for a kid between 5 to 8 years old to 13 years old to learn how to program.
Evolution of Static Config
This is the Google Trends result for the search term, infrastructure as code. On the vertical axis, what you’ll see is plotting the frequency of the search term, where higher on the plot is more frequent. Then, on the horizontal axis is the search term over time. What this is telling us is infrastructure as code is how the industry as a whole is coalescing for how to manage infrastructure. Infrastructure as code generally looks something like this.
Usually, the code is YAML. You write a bunch of YAML, and then you check it in into GitHub or some other source code repositories, SVN system. You check that in, and then you provide that. You apply it to some engine somewhere that takes this, and then takes what you’re declaring, and then, you get infrastructure. This is a made-up example. I think if you’re familiar with infrastructure as code, this might look similar to the things out there. Here, we’re declaring, I want two machines. I want a machine in the us-west region. I want a machine in the us-east region, and here’s all the properties about the machine that I want. Here’s the environment variables that I want. I want this many CPUs, this much memory. Here’s how to configure the healthcheck.
This has been amazing. This is something that has allowed us to really improve the velocity of changing infrastructure over time. It’s simplified deployments quite a lot. One of the things that comes with this is, eventually, your configuration will grow in complexity. It can grow in complexity in two ways. It can scale in complexity of logic. It also can grow in complexity of how much configuration that you’re describing. We’ve already started down this path. We have this format, it’s YAML. We have this problem that we need to somehow scale our system.
Often, what you see is something that looks like this. We have our engine. How do we solve this? We have YAML. Let’s go ahead and go with YAML. Let’s come up with a new property called $$imports, and this accepts some relative path, env/prod.yml. We’ll just look for that file with that path, and we’ll load it in, and then we’ll have our custom rule for how to merge that into our config. Then we can follow this line of thinking.
As more requirements come up, we can add more to our system. Maybe we need to create some parameters for our system. We’ll have $$params. Then this also accepts some relative path, and this is how you create different machines based off of different things. Then maybe we need to differentiate the number of CPUs, depending on whether we’re deploying to production or some other environment. We could keep following this path and create more rules entirely within YAML. The thing about this is, if you take a step back, you start to realize this is actually a programming language. We’ve just invented a new language for our system. It just happens to be one that’s hard to understand, hard to write, and it’s bespoke to this one system.
I want to propose that this is driving in the wrong direction. As you evolve down this path, you create things that are ad hoc. My product describes complexity this way, but another competitor might come out with their own way to manage complexity. As a team that uses both products, they’ll need to context switch between one thing and another. This causes a lot of mistakes to be made easily. For example, as far as the language is concerned, you’ve invented a new language, but the core underlying language is still YAML, and YAML doesn’t know anything about what you’re doing here.
One of my examples is, there was this interpolation syntax with these brackets. As far as the editor is concerned, if you’re in VS Code, you’re in IntelliJ, it’s just a string. It’s also hard because I need to context switch how I manage the complexity from one thing to another. Another thing that’s hard is, as I describe my configuration, I want that to be valid. Because we’re describing data, that data needs to be valid. If I say that the CPUs that I’m deploying is 10 million, then we have a problem.
What Is Pkl?
There’s a great quote from Brian Goetz, who is one of the Java architects over at Oracle. He tweeted this saying, every declarative language eventually becomes a terrible programming language, just without the aid of actual language design.
Just to clarify, this is talking about static declarative languages. We want to flip this around. What if instead of starting with YAML, or JSON, or some ad hoc format, what if we start with a language? What if we build this language with solid principles that let you catch errors, that lets you build abstractions easily, and then you use that same language to describe the data? We found that this works really well. That’s what Pkl is. Pkl is two things. It’s both a programming language and a configuration language. You’ll see what I mean later when I go into my demo. It’s a programming language in that you have all the same facilities that you have in a typical language. You have functions, type annotations, you have imports. It’s a configuration language because that’s what it’s meant to do. It doesn’t have things that are scoped beyond configuration. For example, it doesn’t have an event loop, so you can’t have async/await APIs, or something like that. That’s the overview of Pkl.
Demo
Then, I think the best way to learn about a new language is to see somebody writing it. That’s what I’m going to do. This is going to be a live demo. I’m switching over to IntelliJ. Here, I have an empty file. Before I get into this, we have IntelliJ, but we also have an LSP. We also have plugins for VS Code, for Neovim. Then, if you have some other editor that you like, if it supports the LSP, then you can get Pkl support in there as well. Going back to this. This is an empty Pkl file. This is tour.pkl. I have the Pkl plugin installed, and this is an empty object. You can think of Pkl files, if you use the JSON analogy, as an object that has an implicit open curly brace, an implicit closing curly brace, and then you declare properties inside.
For example, I can say foo = 1. Then, I have my shell open right here, and I can go ahead and eval tour.pkl, and I’m going to produce this in JSON. Pkl is a language that evals into a static format. That’s just one of the things, but it does more than that, but I’ll get to that. I can have this output in XML. I can have this produce YAML and plists. Then you can also extend Pkl to produce other formats too. We have foo = 1. I can say bar = foo. Foo is not just a property. It’s also something that could be referenced. Foo is 1, bar is foo. Let’s go back to JSON. Now we have foo and bar that are both the same thing.
Then, you can have nested objects. Here’s a nested thing, prop = 1, and now we have a nested thing. One of the concepts that drives how we think about Pkl and how we want to design Pkl is to have it closely model the target configuration so that when you read a Pkl file, it looks like the thing that you’re targeting. If you’re describing Kubernetes, for example, you don’t have to guess what that means.
Now I want to jump into what it’s like to use Pkl. Earlier in my slides, we had this. We had my made-up infrastructure as code system, and I want to show you what it looks like when you use Pkl to describe this rather than YAML. One of the concepts of Pkl is you want to describe the schema of the system. Then, once you have the schema, then you can provide the data. Let’s go ahead and do that. I’m calling this FooBarSystem.
Then, let’s take the YAML and let’s figure out what it looks like as Pkl. We have machines, so that’s a property, and we can go ahead and declare a property. In this case, instead of providing a value, because I’m describing the schema, I’m just going to provide a type annotation. Machines happens to accept a YAML sequence of objects, and in Pkl, the way that you describe that is to say this is a listing. This is a listing of something, in this case, let’s call it Machine.
Now that we’ve described this top-level machines thing, the editor has told us, I don’t know what machine is, and then we get some editor hints in here. Let’s go ahead and fill it out. Machines is also an object, so we’ll call that a class, and the region is a String. We can actually do a little better than that. Here, let’s assume that region is a closed set of strings. We only have so many regions. We don’t have us-west-3. We don’t have us-south or us-north.
In this case, we can just say this is us-west or us-east. This might look familiar if you come from TypeScript, or maybe Scala 3, or other languages that have union types. Then we have environment, and these are environment variables. This is an arbitrary map of string to string. In Pkl, we’ll say this is a mapping of string to string. Next, we have CPUs, which is an Int. Earlier I said, maybe it’s not valid if we want to create a machine with 10 million CPUs. We can actually say this is an int, or this is less than 64. What I’m doing here is I’m creating a type that has a constraint on it, and this constrains the set of possible values that you can provide to CPUs. Also say this is a UInt, because maybe negative CPUs doesn’t make sense either.
One more example. The next thing is memory. Memory, over here, is a string. In Pkl, we can do a little better than that. We can say memory is a data size, and data size is a primitive that’s built right into Pkl. What this means is you don’t need to guess how that system represents 4 gigabits, because you don’t write a string, you use the primitive that’s built into Pkl.
I’m going to skip ahead, and I’m going to show you what it might look like in an actual code base. Typically, you would take a little bit of time and make this somewhat polished so that when you later use this, it’s clear to your developers what all of these things mean. I’m going to go ahead and copy and paste that to here. Then I’m going to blast through the rest of this real quick. We have CPUs. Here, I said it’s a UInt8. Then we have healthchecks. Healthchecks has a port, that’s a UInt16. The type, again, it’s a string literal union. Then the interval, on the YAML side, we have 5. What is 5? What does that mean? In Pkl, we can say this is a duration. Again, that’s a primitive type within Pkl. Then, finally, we’re going to say the output of this module is YAML.
Then when you encounter a data size, this is how you should turn that into YAML. When you encounter a duration, this is how you should turn this into YAML. In this case, a data size is turning into a string. Duration is turning into an int. One of the analogies that you can use to think about Pkl is when we defined this, we’ve just defined a form. Then when we define the data, we’re filling out that form. Let’s go ahead and do that. Here’s another Pkl file, myConfig. It starts with amends FooBarSystem. Amends is the secret sauce for a lot of how Pkl works. Amending says, I am an object that is like this other thing except with more things. In this case, the other object is FooBarSystem. Let’s go ahead and fill out the form that we just defined. This has a top-level property called machines. As I fill this in, I know what everything means. I want to declare two machines. The first one, the region is us-west. The environment has HTTP proxy. I’ll go ahead and copy and paste. Then we have CPUs, which is 4. If we said -4, then we get an error.
Constraint violation, I expected this thing, but you provided -4 in this. This blows up on you. One of the great things about doing it this way is you get immediate feedback. Not just for this is an int and it’s supposed to be a string, but this is supposed to be in between 0 and 64, or 0 and whatever. You get refined validation errors built right into the language. Then the memory is 4 gigabits. The healthchecks, I want one healthcheck where the port is 4050, type is TCP, the interval is 5, except 5 doesn’t make sense in this case. We’ll say healthcheck every 5 minutes, which is actually a crazy healthcheck number, but we’ll go with it.
Then, now that we have this, I can use the Pkl CLI and I can eval this, and I end up with almost exactly the same YAML. I have one more machine to go. This other machine is almost the same. I’m just going to go ahead and copy and paste it. The only difference is the region is us-east. Then the environment variable is also calculated in terms of the region, in this case. We’ll eval this again. Now we get the same input. These two YAML files are structurally the same. This is what it looks like to write it in Pkl.
We can actually do a little bit better than this. Like I said, both the us-west region and us-east are almost the same thing. Again, we can use amends as a way to simplify and help ourselves a little bit. Because when you amend, you say, I’m like the parent object except with these things. In this case, I’m going to go ahead and create a new property. Here I’m saying it’s local. Creating a new local property of type machine. I don’t want the region here because it differs from my two downstream things.
Then, the environment variable is also different. In this case, we can actually already use it. Later when we amend it, we render in terms of what we’re amending. Go ahead and do that now. The syntax says amends, but this is an amends expression. When you have baseMachine wrapped with parentheses and then an object block, you’re saying this object is just like this guy except the region is us-west. We can do the same thing here. This object is like the same guy, except the region is us-east. We’ll go ahead and eval and we get the same error. I want to show what it looks like to make an error. I’m going to make an error here. If I make an error, then I get some helpful message saying, this didn’t make sense and here’s where the error is. That’s part one.
We’re going to move into a real-world scenario. In this case, I’m going to show what it might look like to use Pkl to configure something with Kubernetes. Here’s part two. Within part two, I have a file called pkl:Project. I’ve declared two dependencies. These dependencies are packages that are published to pkg.pkl-lang.org. You can create your own packages and publish them at will and that works super easily. I have a pkl:Project with two dependencies, Kubernetes and Prometheus. Then here, again, I’ve defined the schema for my configuration.
In this case, the schema has Prometheus as a property to fill in. We’ve defined some things that are defined in terms of Prometheus. For example, we have a configMap and we have deployment. Prometheus is a bunch of things, but one of the things it does is it’s a scraper that you can deploy somewhere and it can scrape metrics and send it off to some server. One of the ways that you can deploy a scraper is you create a deployment and then you create a prometheus.conf that configures the scraper.
Then you deploy it, apply it to Kubernetes, and then you have your scraper running. That’s what we’re doing here. We’re creating a configMap and a deployment. We’re defining the configMap in terms of Prometheus. The configMap’s data has prometheus.conf, and the value is the textual output of Prometheus. Think about what you would do if you were defining YAML. If I were to do the same thing in YAML, I would again create a configMap with a prometheus.conf. Then, in YAML, I just have a string. Then within the string, the editor just says, this is a YAML string. You lose context there, you don’t know that this is a prometheus.conf. In Pkl, we can go the other way around. We can define an abstraction that says, if you want to deploy Prometheus, this is what you define, you define the Prometheus configuration and then I’ll take care of everything else for you.
We’ve defined the form, and now we can fill out the form. Here’s, again, another file. It amends Prometheus deployment, just like we did earlier. Then we can start filling in the form. In this case, we are creating a Prometheus scraper. This needs some scrape_configs, and we’ll go with Kubernetes scrape_configs. This is of type Listing of KubernetesSdConfig. It’s a listing, and so I need to put things inside. Here’s the thing I’m putting. Let’s continue filling this out. What is namespaces? Namespaces is a namespace spec that takes names. It takes names. We’ll go ahead and scrape foo. This is not important. I just wanted to show what it’s like to use Pkl. Compare this to what it would be if you were writing something like YAML or JSON. That would probably look like you having a browser window open and then looking up documentation.
The cool thing is because it’s part of the language, it’s part of the API of this config object, and it could just look it up just like I would if I were writing Java or Swift or something. Now we’ll go ahead and pkl eval this, and we have an error. I didn’t expect this, but I like that this is happening. There is something about my config that’s invalid. What’s invalid about it? The first scrape_configs needs a job name. This needs a job name, and again, this is called foo. Now it works. Like I said earlier, what we’re doing is we’re actually deploying two Kubernetes resources, but as a user filling in this form, I didn’t have to care about that. All I care about is what the Prometheus scraper looks like.
Then, I want to take this concept and go even further. Part three. Now we have this concept of, we create these abstractions, and we use that to deploy to the external system. Let’s keep going here. I’m going to show you what it might look like to use Pkl to deploy. We’ll stick with the Kubernetes theme, and we’ll stick with Prometheus. How you might use Pkl to manage a large-scale Kubernetes deployment. With Kubernetes, often what you do is you don’t just deploy to a single cluster. You deploy to different clusters all over the world.
Part three, imagine that that’s the root of a repo, and within part three, I have top-level directories called production and staging. Within production and within staging, you have us-east and us-west, and what we’re doing here is we’re using the directory structure to manage the complexity of our config. Here’s the same abstraction. Again, it’s a Prometheus that you fill in, I’ve added a little bit more things here. I’ve added resource requirements and the version of Prometheus that we’re deploying, and then we end up deploying exactly the same thing. This is how you might use this template, this form that I just defined. I do exactly the same thing. I create a file that amends it, and then I can fill it out. In this example, this is a Prometheus that’s deploying, but the difference is it’s deploying version 10.
Then I can also say, in us-east specifically, we want to change Prometheus in this way. Because we’re using the directory structure in multiple files, we could put the things that have to do with production in us-east in this file specifically, and we could put things that have to do with us-west in us-west specifically. Imagine you’re a team, you’re doing red-blue deployments or something, and you want to deploy version 10, deploy that and have that go up for a little bit of time, then you just come here and you say version 10, and then you apply it.
Notice, this file starts with amends “…”. Amends, again, is the thing where it means, I am an object like this guy except with these qualities. In that case, this guy, what “…” means is the first file with the same name in the directory ancestry tree, so it just goes up until it finds another file with the same name. You will go ahead and follow that, and that comes here. This is another file that also amends “…” and this says the resource requirements for all of production should have 10 CPU requests and 8 gigabits for memory.
Then, if I wanted to change something that affects all of production, then I just add that here. As you separate things into multiple files, then it becomes really easy for you to figure out, where is that complexity managed? Does it have to do with production? Then I just go into production and / whatever. In this layout, I have Prometheus that extends component, and component, in this case, is a building block to build a logical set of Kubernetes resources, so another component could be deploying Redis, for example. It could be deploying your bespoke application. This model lets you come in here, create something that extends component, and just define the knobs that you care about for that thing. Then, again, we’ll go ahead and eval that, and we get the same thing. We once again get YAML. If I wanted everything that had to do with production, then I can shell glob that, and I get all the production stuff. That’s part three.
So far, what I’ve talked about is how you can use Pkl to target external configuration, and that’s just one of the ways that you can use Pkl. If your external system doesn’t know how to speak Pkl, then you can use Pkl to render a format that that thing speaks. For example, Kubernetes doesn’t know anything about Pkl, but that’s ok, because we could just render that into YAML, which Kubernetes does understand. However, we also provide libraries for languages. If you’re creating an application, you can think of Pkl as just a library. We maintain libraries for Swift, for Go, for Java, and for Kotlin. Then we have an extension point, and we have an amazing community that’s provided a bunch of bindings for a bunch of different languages out there. I’m going to go ahead and clear that. I want to show what it might look like, I’m going to pick on Java in this case. If you’re a Java developer, this might look familiar. I have a build.gradle. Then, within the build.gradle, I have the Pkl plugin.
Then, I’m using the Pkl plugin right here. javaCodeGenerators, this is interesting. Within this application, I have source main resources and source main Java. If we go into source main resources, I have yet another template, but in this case, this describes the config of my application. Here we have defined the host, the port, the databases, and that’s a listing of database connection. Then, I’ve defined a Java code generator, and I’m going to go ahead and run that. I’m going to call gradlew config Classes. I’m running a task that takes the Pkl source code, and it turns it into Java. Now that I’ve run that, I now have Java available. This is the same thing that we’ve just described in Pkl, except it’s Java.
Now, in the actual application, I’m using the Pkl library to evaluate that file as an instance of a Java class, and that’s here, so that looks like this. It uses the ConfigEvaluator, calls the evaluate method, and then it converts the result into an AppConfig. Down here, we’re loading the AppConfig in my main function, and then I’m just printing. This is a demo anyway, so it doesn’t actually do anything except for printing the line, but let’s go ahead and run it. Here we go. This is the result, indeed, it is listening on localhost 10105, except it’s not. What this means is, in Java, you also get type safe config. You don’t need to worry about what the properties are. You don’t need to call .getProperty and cast it to a string, and hopefully it works, and hopefully somebody didn’t misconfigure it, because in Pkl, that’s type checked. If this evals successfully, you get valid data. Then, in Java, you get type-safe accessors, so you have .host and .port, and then if you call wrong property, then you get a compile error.
Participant 1: You end up with an instance of a Java class that contains the actual configuration?
Chao: Yes, that’s just a Java POJO.
Summary
Pkl’s power is it can meet you with your needs as first-party config and as third-party config. Pkl is just one logical application. It’s one program. For example, what this lets you do is you can manage your infrastructure in Pkl, and you can eval the same stuff directly in Java, so you can make sure those two things don’t go out of sync with each other. Because it’s one language that you have to learn, you can manage all the complexity of configuration directly in Pkl rather than spread it out in different places, and that’s what we hope Pkl becomes. We really like it, and I hope you like it too. We’re just getting started.
Resources
If you want to get involved and learn more about Pkl, here are some things for you. Our website is pkl-lang.org, and on there you can learn all about the language. There’s a tutorial that you can go through. There’s a language reference. There’s links to all the libraries and all the things that we do. If you want to get involved in development of Pkl, please do. We love pull requests. We love our contributors. It’s at github.com/apple/pkl. Then, we also have an awesome community of users already. You can go to pkl.community. This is not managed by us, by Apple. This is maintained by other people. On there, you can find a Discord, and some of the maintainers hang out on there too.
Questions and Answers
Participant 2: These Pkls are really well-suited, I think, for distribution throughout an organization, is there any mechanism that is supporting that? For instance, you don’t want to copy files, but you want to make maybe reference to a library that contains any Pkl files.
Chao: Earlier I showed a pkl:Project file with dependencies. Those dependencies are called packages, and you can create your own packages and publish them anywhere you want.
Participant 2: There is no particular format?
Chao: There’s a format. You’d use the CLI to create packages.
Participant 3: I’m just curious if you could talk about some design goals of this compared to CDK and the AWS Constructs library, because it feels like there’s a lot of similar goals, but I’m sure that you have some different ones in mind as well.
Chao: CDK is focused on a particular use case. CDK is meant for describing infra. That’s AWS?
Participant 3: CDK is built on a library called Constructs that’s more general purpose.
Chao: Maybe the bigger question is, I’ve described these things where you can use a programming language for config, but there’s also libraries that use Go, or Python, or TypeScript as a DSL for config, why would you use Pkl instead? I think one of the reasons that you would use something like Pkl is because if you have a polyglot organization where you have developers that use Java, that use other things, and you try to convince them, to configure Java, why don’t you use the CDK in Go? That’s going to be a hard sell. I think another thing is, unlike Python and TypeScript and other languages, Pkl’s designed for config, and so it has a lot of things that are lacking in those languages. It’s purposefully lacking things that are available in those other languages. For example, I showed type constraints. This is a port that should be between 10 and 63. You could describe that in Pkl, and the type system understands that, whereas TypeScript doesn’t.
Participant 4: I think you talked about publishing packages and having a repository. Are there well-known packages to define things? I know I’ve had to deal with Envoy config, for example. Is there something that predefines all the types that I can just reference and then navigate through and understand how to configure it?
Chao: We maintain packages, and we have a doc site that you can go through to look at all the packages that we maintain. Then, we also have code generators that take, for example, JSON schema. If you have schema already written in JSON schema, you could just generate Pkl from that.
Participant 5: I’m just curious if you can talk a little bit more about something like Pulumi, and what are some reasons why someone would use Pkl over that?
Chao: I think that relates to the first question, which is, why would you use Pkl over Python or TypeScript? Which is, Pkl is a language designed for config, and it has a lot of features that don’t exist. Then, it’s also a lot more portable. If you’re a Java developer, you’re probably not going to want to use Pulumi to configure your Java app. Pulumi is also multiple things. It’s like SDKs plus the Pulumi engine. It’s apples to oranges. You can use Pkl to configure Pulumi too, because they have a YAML spec.
See more presentations with transcripts
