Nonces: The Other Problem With WordPress Caching

Cloudy mountain top

Nonces are one of those things that, hopefully, all WordPress developers know they need, but there is still a lot of confusion about. Maybe it’s beacuse “nonce” means a number used once and a WordPress nonce is not a number and it can be used. So it’s closer to a nonce than a nonce word, but it can be used multiple times, but I digress.

The point here isn’t semantics, it’s to discuss some problems nonces present. I run into problems with Caldera Forms + caching plugins that are set with infinite cache lengths as this causes our nonce validation to fail.

This article isn’t about setting up caching plugins or services or which one is best. It’s about understanding one of the gotachas with many caching plugins: caching nonces.

Nonce Problems

I want to talk a bit about what nonces are, why we use them and what can go wrong. I’ll keep it brief, as I’ve written a more detailed article on nonces for Torque already.

What Is  A Nonce?

As I said earlier, “nonce” means “a number used once.” For example, most payment processors, like Stripe issue a one time use token, which is a mix of numbers and letters, for creating a payment. They call it a token, but same concept, technically a nonce word. You request the token with a public API key and then complete the payment with the token and the secret API key.

Nonces are not a security feature by themselves since they become public once printed to the page. Also, in WordPress, nonces can be used more than once.

Nonces help prevent cross-site request forgery, beacuse they can be used to prevent requests from external sources form mimicking intended HTTP request. Also, nonces make up part of your strategy to prevent XSS attacks since they help verify that the request is coming from the intended user.

Nonce Expiration

Snowy mountainAs I said in the last section, nonces can be used more than once. They remain valid for a set period of time. By default this is 12 hours, but can be changed with the nonce_life filter. This design decision means that WordPress doesn’t have to keep a record of nonces created and if they have been used. That probably would have required a new database table.

This is important to keep in mind. As are the other ingredients of a nonce. For example, one of the components that goes into a nonce is user ID, or 0 if the user is not logged in. This prevents nonces from non-logged in users from validating when a user is logged in. This by the way is why you should always send the REST API nonce on every request. If you don’t other nonces might not validate properly.

Forms, Caching And Nonces

Every form in WordPress should use a nonce to prevent cross-site forgery requests. All forms provided by WordPress core do. This includes comment forms. Also, your contact form should. We use a nonce in Caldera Forms.

In the last section I mentioned that nonces can be reused for a period of time, normally up to twelve hours. This is good beacuse it means that you can cache the content of a page with a from and the form will still work, for up to twelve hours.

The “up to twelve hours” part is the rub.

The idea behind a static HTML cache is that you only generate the page output once, and serve the same thing, which should be identical to all other visitors. This makes sense as it prevents running the same database queries and running the same PHP scripts to do the same thing over and over again.

Great, but if you have a nonce in your content, it could get expire and prevent your forms from working. Obviously this is a problem we face with Caldera Forms, when it is used with caching plugins or proxies like CloudFlare. This is also one of the reasons people have moved away from native commenting in WordPress.

What To Do?

One thing not to do is to re-generate nonces from other plugins, before they can be validated. This bypasses the problem, but defeats the purpose of the nonce. I’ve seen plugins do it though 🙁

It’s common practice to bypass the cache for logged in users. One reason is that the content may be unique to the user. The other is that nonces are unique to the user and can’t be shared.

The other thing you really should do is keep your cache times short. Definitely less than 12 hours, but probably even shorter. Also, relying on a non-PHP solution like Varnish or nGinx as a reverse proxy is preferable, but you still need to make sure your cache time is short enough.

By the way, if you’re wondering which caching plugin I recommend, I don’t use one. None of my sites receive thousands of concurrent views. The Caldera Forms site is I’m dealing with few thousand page views a day, not a time and we have it on a small AWS VPS that can handle that easily, especially with all of the assets coming from a CDN.

If they did, I would probably use Varnish and/or nGinx as a reverse-proxy over a plugin. I do use Redis as a persistent object cache whenever possible. A good WordPress host should provide a static HTML cache, powered by a server appliance, with a short cache length.

Nonces Are Important, But Don’t Get Cute

Mountain valleyNonces are important, if you do WordPress development, make sure you read the WordPress developer handbook section on nonces. If you have questions, ask in the comments. Also, be careful not to get cute with nonces, and always use wp_verify_nonce() to verify nonces to avoid issues like I discussed in this post.

 

 

Fun Challenges With Recent Caldera Forms Updates

cropped-CalderaWP_Icon_512x5121.pngLast week, we release a new version of Caldera Forms and a new form translations add-on for Caldera Forms. These were both fun projects to work on and I wanted to share a more technical overview of some of what I did on those projects than I gave in the official release post.

I hope this post is interesting and useful to all WordPress developers, whether you are a Caldera Forms user or not. But if you’re not, maybe it’s time to try it 🙂

https://calderawp.com/2016/11/caldera-forms-translations-and-more/

 

Caldera Forms Translations

Photo by: Ben MoorePreviously, When using Caldera Forms or most other form builders on a multi-lingual site, you had to have to create multiple forms, one per language. That is a pain to manage. Making one change meant editing multiple forms.

Caldera Forms Translations is our new translations add-on. One forms, many languages. I originally was planning this as a feature of the core plugin since we didn’t want to make it a paid add-on. That said, figuring out if a form has translations, then if each field has a translation, and then if that translation is in the right language introduces a bit of overhead to form rendering.

It’s not that much, and its worth the trade-off in exchange for the functionality. But this is one of those features that is great for certain users and not worth it for many, so an add-on makes perfect sense.

The interface for this plugin was a lot of fun to build. It’s actually the first time that we’re using the new Caldera Forms REST API. I will be blogging more about the challenges of implementing the WordPress REST API in a legacy project soon.

I actually did the first pass at this plugin using admin-ajax. I thought it was just going to be one ajax action I need for the admin UI. Turned out I needed three.

You can see the commit where I deleted those callbacks here. But I can best summarize how messy things become validating and authenticating these type of HTTP requests get with this:

if( cf_translate_can_translate() ){
        if( ! empty( $_POST[ 'language' ] ) &&  ! empty( $_POST[ 'form_id' ] ) && ! empty( $_POST[ 'fields' ] ) && is_array(  $_POST[ 'fields' ] ) && ! empty( $_POST[ CF_Translate_AdminForm::nonce_field_name() ] ) ){
            if( CF_Translate_AdminForm::verify_nonce( $_POST[ CF_Translate_AdminForm::nonce_field_name() ] ) ){

That’s terrible and doesn’t even get into sanitization. I think it was the third time I had to refactor that when I decided to add the Caldera Forms REST API infrastructure to Caldera Forms 1.4.4. Previously I had infrastructure and some implementation on the 1.5.0 milestone.
The REST API routes for this plugin has three endpoints. One is for saving settings. The other two are for adding and saving languages.

This add-on supports more than a hundred languages. I don’t want each field to by default to support all of those languages. So, in the UI, there is a selector for languages and an add button. This makes an API call to get the field data, and insert it into the local variable that tracks the languages of the form.

The JavaScript for this interface is a bit experimental and doesn’t use our normal UI framework. I’m playing with new ways to build UI. This add-on was an experiment in using a more structured system. I ended up with a pretty modular system that separates each part of the UI into its own closure. That’s good, because I will probably replace each part one at a time as I continue to use this add-on for experimenting with UI systems and figure out which JavaScript framework we will use in the future.

I will admit that some of this modularity was a bit of overkill and I definitely broke some of my self-imposed rules in order to finish it. Still, I think it would take minimal refactoring to pull one part of the system out if I wanted to replace it with something else, or reuse it elsewhere. Also, it works, which is the point.

One thing I am very happy with is how the class that handles translating fields during rendering works. You can read the source here. This class is a good example of using dependency injection — it takes the form configuration through its parent’s constructor — and using inheritance to create a reusable system. Its parent class is a system for conditionally adding a filter. I might add a similar system to Caldera Forms itself, I’m not sure yet.

Caldera Forms 1.4.4

Photo by: petradr

The new version of Caldera Forms is mainly a bug fix release, with a few enhancements. Most of the bug fixes were to resolve accessibility issues, as we continue to work towards our goal of being the most accessible WordPress form builder. Mainly the new improvements are infrastructure for add-ons and custom development. The REST API infrastructure will get its own post soon.

The biggest improvement in this new version is how file and advanced file fields handle uploads. One of the trickiest thing about building a form builder is dealing with the fact that most form submissions use more than one HTTP request. A validation error means another request may be made. Also, our advanced file field uploads multiple files via multiple AJAX requests and then the main submission happens.

As a result, when the main submission is being processed for a form with an advanced file field the $_FILES super global is empty. These fields allow for saving files to the media library and/ or attaching the files to an email.

If both options are selected, that’s easy. We upload the file to the uploads directory, add it to the media library and then attach that file to the email. When the file needs to be attached to the email, but not saved to the media library, that’s where it gets tricky. The file needs to be saved to the server so it can presit between sessions, but users are right to expect the files to not stay there forever. That creates a privacy issue and disk space usage issue.

My solution for this scenario, which is based on a conversation I had with Micah Wood when we had dinner last time I was in Atlanta, was mainly written in my my car as my wife and I drove back from Atlanta. I introduced a new class to Caldera Forms for handling file saving and implemented it in the existing callback for file fields.

This new class manages files on the server and introduces a concept of a private file. Private files are stored in an almost randomly named sub directory of the uploads directory and deleted later. That dub-directory names is named using a hash that can be created predictably, but only by the server, not an outside observer. As a result I can find the directory later and delete it.

Choosing when to delete that file is also a challenge. It should be deleted after form submission, but it needs to be available when the email is sent, though that email may be disabled. In addition, I had to account for files uploaded to form submissions that failed validation, and were never completed.

So, I wrote two ways to delete it. The first was to set a single CRON to delete the file(s). If form submission is successful, the files will be deleted by the time it runs, but its a good fallback. The other method hooks in at the last action that would be run if no email is set. It checks if the email should be sent. If not, it deletes the files. If an email should be sent, it adds a hook to run after the email is sent, to run basically the same callback.

Read The Source Luke

I hope you found this article useful and that you dig into the source code I’ve shown. Reading the source is the best way to learn.

Writing articles like this is something I’d like to do more of because most of my articles on development are a bit divorced from the real world. That’s fine, I’m teaching a principle most of the time and contrived examples are needed.

The real world is a lot messier than a tutorial. Improving a large code base and fixing bugs without breaking more than 30,000 websites running that code is a challenge. It’s a lot of fun, and its a great way to learn.

 

The WordPress REST API As Connector

Earlier this year I gave a talk on what having the WordPress REST API means. I actually gave the talk twice, once I. Jacksonville and once in San Diego and both times I started by asking why we needed a REST API.

Both times I asked, someone in the audience said that the WordPress REST API would help us display content from multiple sites in one location. I agreed.

Both times I asked, someone in the audience said that the WordPress REST API would help us decouple content management form the front-end. I agreed.

Tomorrow, I will be at WPCampus, a conference for WordPress in higher-ed and this is what I am talking about. I’m talking about using the WordPress REST API to show content from multiple sites is one central location.

I hope this will help solve problems for educational institutions with many sites, managed by different departments, that want to share that content throughout their online presence. But, these strategies will also help organizations of any type solve scaling issues.

In addition, companies with multiple brands can use these strategies to share content between different sites and different brands. In fact, half of this talk is based on code I wrote to solve that problem for my own company, Caldera Labs.

Each of our brands has its own WordPress site. Our central site is a static HTML site, hosted on Github for free. It loads in about half a second, and has no database. Instead, we pull content via the WordPress REST API, using jQuery AJAX from multiple sites.

Photo by: Jakub SejkoraThe WordPress REST API is changing ways we use WordPress and ways we consume WordPress data. I look forward to learning more about how others use this new technology and continuing to teach about how to use it as well as using it.

As I prepare for my 6th WordPress conference so far this year, I’m reminded that WordPress is a connector. Yes, the REST API provides a way to connect WordPress sites together and WordPress to other technologies. But, more importantly, WordPress can help us reach our goals when we have the right tools, and helps connect us to awesome people in the community 🙂

Slides

 

Links

 

WordCamp Orlando: 5 Major Events In The Life Of A WordPress Request

I’m presenting at WordCamp Orlando today on major events in the life cycle of a request to the front-end of WordPress. The goal of this talk is to help you understand how we get from this:

Screenshot of WordPress' index.php
This is index.php, where every WordPress request begins.

To this, or really any page on your site:
Screenshot of Twenty Fifteen

But what I really want to encourage people to do is to get in the habit of reading the source
Read The Source Luke meme

Helpful Links

 

Learn The WordPress REST API With Me

API Wapuu by Michelle Schulp.
API Wapuu by Michelle Schulp.

I’ve had so much fun teaching myself the WordPress REST API for the last year or so. Part of what has made the experience so great is sharing what I’ve learned a long the way — tutorials, WordCamp talks, Podcast appearances, a free ebook.

I can’t decide what I love more — using the REST API to build custom APIs the WordPress way or sharing what I’ve learned. So, I’ll just keep doing both.

That’s why I’m happy to announce I’m working on a four part video course, teaching you how to use the WordPress REST API.

The REST API Is The Future Of WordPress

Actually, it is already in use by those doing the coolest stuff with WordPress.

Infographic on how awesome the WordPress REST API isWordPress just hit 25% market share among content management systems. That’s awesome, the more growth in our industry the more potential customers or clients there are for people like you and me who offer WordPress services and products. We know that there are a lot of negative opinions about WordPress out there. Some of them are fair, some of them are not.

The more times we can exceed the pre-conceived opinions about WordPress, deliver more “You can do that with WordPress?” moments, the more our industry will grow. I want you to be the one to deliver that “wow” moment to your clients. I want you to be able to give more to your clients than they expect from a WordPress site. Don’t forget to raise your rates 🙂

But Wait There Is More!

And more!!

The course will be available in December 2015. But if you sign up today you will get a special discount. Don’t delay — once the course is ready for download, the price goes back up.

Want to learn the REST API one on one with me? I’ve got a special bundle that includes the course and a one on one consult call via Skype or Hangouts.

Recovering Scratch Files In phpStorm

I love phpStorm. It’s the best IDE for WordPress development, period end of story.

One of its many cool features is the scratch files. I used to have a bad habit of typing up a quick test, or example code in whatever file was last open, promising myself to clean up after myself later. Or, to take advantage of phpStorm’s ability to jump to a function, class, hook callback, etc’s source, by command clicking on it, I’d just type in the function (or whatever) and jump through.

Scratch files are the right place for this sort of thing, as while they actually can be executed, they are not executed as part of WordPress. Until recently, I was under the mistaken impression that once you closed a scratch file it was lost for ever. This made Josh a very sad boy many times.

Turns out I was wrong and you can easily get back to any closed scratch file in a project. All you have to do is click on the dropdown at the top of the file viewer pane, and change form project files, the default to scratches. All your old friends are there! Day == saved.

save-scratch

Docs: https://www.jetbrains.com/webstorm/help/scratches.html

Is this me blogging at my self? Yes, yes it is 🙂

Extending The WordPress REST API

Remember the first time you got annoyed WordPress didn’t do something, and then you realized with the right amount of Googling, experimentation and foul language, you could probably make it do that?

I don’t exactly, but some how it got me to where I am today, and I’m pretty happy about that :)

Hopefully, we will soon have a RESTful API in WordPress core. Since this is WordPress, the features of the REST API are sensible defaults, designed to work on tens of millions of sites. That means that there will be plenty of cases where the default routes do not fit the exact needs of a site, app, plugin, or theme.

Of course, since this is WordPress, the REST API is highly extensible. I’ve now written two articles on extending the REST API for Torque, and contributed to the docs on this subject.

I’d like to give you a brief overview of how to extend the default routes, and add your own, and show you some examples of plugins I’ve created to do  both. In addition I’ll link to the Torque articles and documentation to help you learn more.

But First

Version two of the REST API is on WordPress.org!

Version 2 of the REST API can now be installed as a plugin from WordPress.org. This makes testing easier. The two versions are separated so those relying on version 1 can continue to do so.

In the future the plugin will provide backwards compatibility for version 1 routes and endpoints, once version 2 has been merged into core. At this time, backwards-compatibility is not complete to version 1, and anyone relying on version 1 should continue to use the version 1 plugin.

The oAuth1 plugin from the REST API team will be available on WordPress.org soon.

Extending The Default Routes

add_filter( 'the_content', ...

WordPress ships with a really comprehensive set of URL mappings, that are all most sites need. But that doesn’t mean that an off the shelf theme, shows all the data you need. That’s why we modify template files, or hook into the_content to show custom fields and other non-standard data.

The REST API comes with a really useful set of default routes and endpoints. But those default endpoints are not always going to show the data you need, or give you the ability to update all the data that you consider part of the post/ term/ user / comment.

The REST API has a really helpful function called “register_api_field” to add the ability to read and/ or write new fields, from custom fields or any other source to an existing endpoint.

I covered using register_api_field, as part of my article for Torque on working with post meta data via the REST API. I actually turned the code examples in that article, showing how to read and write SEO title and description fields via the API into a plugin, which is in beta now, and we will be released at CalderaWP soon.

I wrote a few more examples of using register_api_field in the documentation site for the API. The version 2 docs are coming along nicely BTW. If you are looking for  away to contribute to this project, docs are a great, and needed way to help.

Rachel Backer, co-lead developer of the REST API project gave some examples of how this cool function works during her recent WPSessions presentation, which I strongly recommend watching.

 

Add Your Own Endpoints

$query = new WP_Query( …

The WordPress REST API is not just at the default API routes and endpoints, it’s infrastructure for creating APIs. That infrastructure is really well done. As someone who has rolled my own API on many occasions, I look forward to the day, when I’ll be able to take all of this API infrastructure for granted.

Custom endpoints are really useful for when you don’t actually want to query with a core query tool (WP_Query, WP_Users, etc.) but want to wrap some other function, or class. I’m particularly fond of the fact that I can now create classes for fetching and saving data, and then easily wrap an API endpoint around it.

On thing I love, about custom endpoints is that the API establishes a pattern of separating sanitization, validation, permissions checking, processing and responding into discrete methods in your class. Also, as this becomes the standard, hopefully we will see less misuse of admin-ajax in the front-end.

After writing about creating custom endpoints for the REST API in Torque, an article based on the CalderaWP API, I created an add-on for SearchWP, that allows for using that plugin’s awesome search powers via the REST API. After reading my article, I recommend reading documentation for adding endpoints. It explains how to follow the core API patterns and provides some good, generic example code.

Needs Help Figuring Out What To Do?

I know, it’s a lot

This can be a lot to digest. If you’re getting ready to jump into a project involving the REST API, you may not be sure if you should extend the existing endpoints or add your own. It’s a tough question, and the answer varies from project to project.

If you need help preparing for this type of project, I offer consult calls to sort through these types of questions and provides guidance moving forward. I also love doing custom development with the REST API, especially when I get to write custom endpoints.

Let me know how I can help :)

Access Control Headers For The WordPress REST API

Sending headers, including cross-origin (CORS) headers has changed a bit in version 2 of the WordPress REST API. Access control headers are sent by the function rest_send_cors_headers(), which is hooked to rest_pre_serve_request. You can easily change the headers by unhooking that function, and adding your own.

Below are some examples using access control headers, but really any type of header could be added here. That said, keep in mind that the class WP_REST_Response, which should be used for all responses, also gives you the ability to add headers. Any headers unique to a request should be set there.

/**
 * Use * for origin
 */
add_action( 'rest_api_init', function() {
    
	remove_filter( 'rest_pre_serve_request', 'rest_send_cors_headers' );
	add_filter( 'rest_pre_serve_request', function( $value ) {
		header( 'Access-Control-Allow-Origin: *' );
		header( 'Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE' );
		header( 'Access-Control-Allow-Credentials: true' );

		return $value;
		
	});
}, 15 );


/**
 * Only allow GET requests
 */
add_action( 'rest_api_init', function() {
    
	remove_filter( 'rest_pre_serve_request', 'rest_send_cors_headers' );
	add_filter( 'rest_pre_serve_request', function( $value ) {
		$origin = get_http_origin();
		if ( $origin ) {
			header( 'Access-Control-Allow-Origin: ' . esc_url_raw( $origin ) );
		}
		header( 'Access-Control-Allow-Origin: ' . esc_url_raw( site_url() ) );
		header( 'Access-Control-Allow-Methods: GET' );

		return $value;
		
	});
}, 15 );


/**
 * Only allow same origin
 */
add_action( 'rest_api_init', function() {

	remove_filter( 'rest_pre_serve_request', 'rest_send_cors_headers' );
	add_filter( 'rest_pre_serve_request', function( $value ) {
		header( 'Access-Control-Allow-Origin: ' . esc_url_raw( site_url() ) );
		header( 'Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE' );
		header( 'Access-Control-Allow-Credentials: true' );

		return $value;
		
	});
}, 15 );


/**
 * Only from certain origins
 */
add_action( 'rest_api_init', function() {

	remove_filter( 'rest_pre_serve_request', 'rest_send_cors_headers' );
	add_filter( 'rest_pre_serve_request', function( $value ) {

		$origin = get_http_origin();
		if ( $origin && in_array( $origin, array(
				//define some origins!
			) ) ) {
			header( 'Access-Control-Allow-Origin: ' . esc_url_raw( $origin ) );
			header( 'Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE' );
			header( 'Access-Control-Allow-Credentials: true' );
		}

		return $value;
		
	});
}, 15 );


I have a plugin for setting CORS for GET requests and a plugin for setting CORS for ALL requests on Github.

Read The Source Luke

There was a time when I really wanted to work for 10up, not to be confused with the time they wanted me to work for them. Before I interviewed with their president and founder Jake Goldman, I did as much research on him as I could on him.

That’s a good pre-job interviewing strategy. It didn’t work for me then, but I learned some really useful stuff from reading up on Jake and looking through his WordCamp presentations.

My point here is not to present some theory on getting jobs that is likely full of shit. I hear 10up is hiring and if you want to work for them, awesome. I don’t have any advice besides contribute to open source projects as much as possible. Actually, that’s my advice on getting any job in the WordPress ecosystem.

One of the best pieces of advice I got from reading up on Jake was that when you don’t know how to do something, you should read the source code, before you go to the codex. This is one of the most useful bits of advice I’ve ever gotten and I repeat it all the time.

Why You Must Use The Source

Search your feelings, you know it to be true!

Read The Source Luke memeSince you do all of your development locally — you do right? — you probably have  like 14 copies of WordPress on your computer. WordPress is super well documented inline and the code is designed to be readable. There is a lot of abstraction that could be done in WordPress core that would come at the expense of readability that I am happy has not been done.

I’m not here to knock the Codex, or the code reference at https://developer.wordpress.org/reference/. They are both great resources, and I use them.

What you can only get from reading the source is an understanding of how something works. The documentation is a reflection of what it does, or should do. The source, along with a quality inline debugger tells you what is happening — you are using xdebug right?  And along the way, you will learn a ton about development patterns by reading the source.

It also means that if your usage of the function/ method/ hook, etc. doesn’t work as intended, you are in the exact right spot to start asking why. You can see what should be happening and then start debugging to see what is happening.

Most importantly, documentation doesn’t always exist. I wrote an article recently for Torque on adding custom endpoints and routes to the WordPress REST API. At the time it was published, I’m pretty sure my article was the only documentation on how to do it.

The lack of documentation wasn’t a problem for me. I just read the source for the default post control. You can’t allow yourself to go through life being rendered powerless by a lack of documentation.

It’s A Free Software Ya’ll

freedom[1] = The freedom to study how the program works, and change it so it does your computing as you wish.

If you want to be a creative non-fiction writer, you’re going to need to not just read, but critically study Truman Capote, June Didion, James Baldwin, Hunter S. Thompson and others top non-fiction writers. That’s a huge part of learning to write in that genre that you can’t skip.

Being a developer is not different. Go grab a well written plugin, and figure out how it works. Along the way you will learn more about how to work with that plugin than you will be reading documentation, no matter how well documented it is.

Pull up the file in WordPress with the API you use the most and trace how it executes the code your throwing at it. Next time something goes wrong with that API, you will know exactly where to start your search for the cause of your issue.

It’s a free software, don’t forget to use your freedom to learn how the program works and to document your code to help others learn from what you do: