Data Sync Series - Part 3

This is the third blog post in my new blog series on data syncing in an app. Links to all of the posts in the series are at the bottom of this post.

So. I’ve got an idea for an app. It’s not a great idea. It’s not original. It won’t make me a penny. But I think it will work well for this series on data syncing.

My daughter gave me the idea for a shared family grocery list. She wanted fruit snacks from the grocery store. I kept forgetting to pick them up, and she said we should have a shared list for groceries. I know there are other apps out there for this, but I needed an idea for this series.

The App

The app will be a series of lists. You can use them as TODO lists, or shopping lists, etc.

Lists can be either private and shared. Private lists will be your own thing that no one else can see. Shared lists will be able to edited by multiple people that you’ve shared the list with.

Here’s a quick wireframe of what this simple app may end up looking like.

There’s not a lot going on. There doesn’t really need to be. The important part of this app is that there is data to be synced between devices and shared between users.

More App Details

The app will be written entirely in Swift 4 in Xcode 9. The deployment target for the app will be iOS 11. There’s really no need to work with anything else in this project.

I’ll be using CloudKit and either Core Data or Realm. I haven’t fully committed to Core Data or Realm as the persistence model. I’ve used CoreData for numerous projects in the past. We use Realm at my day job, I would mind learning more about it.

This will be an app for both iPhone and iPad. These days that should really be self explanatory, but I thought I would mention it anyway. Depending on how things go, I may make it available for the Apple Watch too.

What’s Next?

The next step is for me to start building this thing. Once I get something very basic in place, I’ll be publishing it to GitHub in my repo for this project calls listsApp.

Notes:

I used this Sketch Resource to build the wireframe above: Mobile UI Wireframe Kit Sketch Resource

Part 1 | Part 2 | Part 4

Data Sync Series - Part 2

This is the second blog post in my new blog series on data syncing in an app. You can read the first one here. This took me a little longer than I wanted, I got busy with the kids and end of year school stuff for them. Sorry for any delay.

In this post, I’m going to take a look at the various types of data synchronization that exist and outline which one I’ll be using in the new app that I develop.

A lot of the data synchronization technique information listed here is taken from this excellent article on Objc.io called Data Synchronization by Drew McCormack. He does a great job of explaining this topic. Some of the example services in his post no longer exist, but the concepts are solid.

Synchronization Types

At a very high level, there are two types of data synchronization. Both of the synchronization types can be either synchronous or asynchronous. This technically makes four types of synchronization methods. The two types are Client-Server and Peer-to-Peer synchronization.

Client-Server Synchronization

The client-server approach means that there is a server which acts as the source of truth. Not only does the server act as the source of truth, it also has some “smarts” built in to handle any conflicts that may come up.

In many of the applications that I have developed as an iOS Engineer have used a Synchronous Client-Server model. The server (or set of servers) act as a source of truth. The clients request the latest data from the server. But any changes to the data are immediately sent to the server for processing.

An Asynchronous Client-Server model works just like the synchronous version, except there is an in-app (local) copy of the data. This is usually a third party solution that gets integrated into your app. Any third party solution here, takes care of syncing with a server. The application’s interaction with the data should remain the same as the synchronous version of this model.

Peer-to-Peer Synchronization

Peer-to-Peer synchronization means that there isn’t a “smart” server in place, the syncing and any conflict resolution is handled by the clients.

Synchronous Peer-to-Peer is old school. It was used largely for syncing things like devices (iPods, etc). Clients (or peers) communicate with each other synchronously.

In an Asynchronous Peer-to-Peer model, the clients communicate with each other, figure out the most accurate data (or “truth”) and deal with any conflicts that may arise. The only data that is often passed between clients (and “the cloud”) are transactional logs, or what’s changed in the data.

Conflict Resolution

I mentioned conflicts a few times earlier. What are they? A conflict is when the same data is changed in two different locations. The server or clients, need to know how to resolve the conflict. There are a few ways to do that, but I’ll save that for another post.

Our Synchronization

For the exercise that I outlined in my previous post, I’ll be using an asynchronous peer-to-peer model. I think that this model is ideal for iOS (and other mobile) clients. This allows for the most flexibility in a (still) questionably connected world.

The cloud storage mechanism we’ll be using is CloudKit. In the past, I’ve wanted to use Core Data iCloud Sync. Using CloudKit isn’t a huge departure from that desire to stick with Apple technologies. CloudKit has been reported as a solid technology.

Wrapping Up

I did not go into great detail about the various synchronization types. Drew already did a very good job of that in his article. I wanted to give much briefer overview.

This is how I understand these syncing types. If I’m wrong, or misguided, please feel free to let me know. I’ll be happy to make any corrections.

In the next post, I’ll be discussing the app itself. I have the rough idea for a new app in my mind. The details haven’t been completely worked out yet, but I’m OK with the details changing.

Soon enough the real work (and fun) will begin.

Part 1 | Part 3 | Part 4

Data Sync Series - Part 1

I’ve been thinking about incorporating a data syncing component to my app MyCntdwn for a long time now. I have just never gotten around to it. It’s always been a daunting idea.

When I first wanted to add data syncing to MyCntdwn, it was 2012 and Core Data iCloud Sync was a thing. It was also a mess. I had a few false starts using Core Data iCloud Sync. I even spent time with Apple Engineers at WWDC and still couldn’t get it working right. I eventually gave up on it. I’ve heard that Core Data and CloudKit have made things much easier to get working, but I don’t have any experience.

My first experience with data syncing in 2012 has really scared me off. I’ve thought about it from time to time, but never really committed to it.

It’s time. It’s time to finally sit down and figure out how to do this. I’m going to do things a bit different here. I could just spend the Spring/Summer figuring out how to do data sync, add it to my app and call it a day. Instead, I’m starting this series.

This series is going to be an experiment. I plan on writing a completely new app, open-sourcing it, and documenting the entire thing. This could be a huge failure. I could get three or four blog posts into the series and give up. I hope that doesn’t happen. I hope that I can follow through on this idea. It’s an ambitious project for me.

I have an idea for the project. I’m not sure which technologies I’ll use for syncing. I believe I have several options for the syncing technology, but that’s something I’ll weigh in a future blog post.

Part 2 | Part 3 | Part 4

Charles Proxy Primer

I’ve been an iOS developer (professionally) for over six years. There is one constant in every iOS app that I’ve written for someone else. They all communicate with a web service to read and write data from the iOS app/client. Even my personal side project apps are moving from completely isolated on the device to talking to a web service.

If you work on any sort of mobile client that communicates with a web service, you should learn how to use Charles Proxy. I’ve been using this tool for years and find it absolutely indispensable.

This post will give a primer on using Charles with a simple iOS project. I wrote a very simple project that will load the Curated Images from Unsplash. You can grab the source code from GitHub.

All of the setup in this post is for the iOS Simulator. But setting up Charles to talk to other devices (tvOS or real iOS/tvOS devices), is easy enough. I have written a separate guide to setting up Charles Proxy for an Apple TV device, which you can find it here: Setting up Charles Proxy on Apple TV (tvOS).

Please Note: The way I’ve outlined the instructions here is not the only way to do things. Feel free to explore the app on your own.

Also Please Note: I’ve included short YouTube videos showing the steps for the instructions below. Some of the instruction sets can be complicated and the YouTube videos should help out.

Charles Proxy Details

Before we get started, here are some details about Charles Proxy. Directly from the developer’s website:

Charles is an HTTP proxy / HTTP monitor / Reverse Proxy that enables a developer to view all of the HTTP and SSL / HTTPS traffic between their machine and the Internet. This includes requests, responses and the HTTP headers (which contain the cookies and caching information).

Website: http://www.charlesproxy.com

Cost: $50 USD

Free trial? Yes, for 30 days

Charles Proxy Overview

Charles Proxy is an HTTP proxy/monitor application. It allows you to monitor web traffic from a variety of sources, including macOS/iOS/tvOS devices and the iOS/tvOS simulator. This means that you won’t have to write temporary code like this:

print("JSON Response: \(JSON)")

or

NSLog(@"JSON Response: %@", JSON)

Instead, you’ll be able to setup your device/simulator to use Charles Proxy and watch the inbound/outbound HTTP(S) traffic without making a single code change.

Setting up Charles

The steps for configuring Charles to watch iOS Simulator traffic have gotten so much easier over the years. Follow these steps to get started:

  1. Launch Charles Proxy
  2. Make sure that you have run the target iOS Simulator at least once.
  3. Quit the iOS Simulator
  4. Open Charles and navigate to Help -> SSL Proxying -> Install Charles Root Certificate in iOS Simulators

If your app is talking to a HTTPS website (as it should be), there may be a few extra steps you need to do in order to actually see the data. By default, the HTTPS/SSL response from a web service may look like this:

Response SSL Off

In order to unscramble that mess, you need to enable SSL Proxying for the domain. Follow these steps to enable SSL Proxying for the Unsplash API domain:

  1. Open Charles and navigate to Proxy -> SSL Proxying Settings
  2. Click on the SSL Proxying tab.
  3. Make sure Enable SSL Proxying is enabled.
  4. Click the Add button at the bottom.
  5. Enter api.unsplash.com into the Host field and click OK.

Thats it. When you restart your app, Charles should start capturing your web traffic. The HTTP response will look more like this:

SSL Overview Enabled

If you’d like to know how to install the SSL Certificates for other devices, the Charles website has instructions on how to do this.

You can use Charles as it is, or you can customize things. I don’t like having all of my web traffic recorded, so I limit what Charles records on my behalf. In order to do this, follow these steps:

  1. Open Charles and navigate to Proxy -> Recording Settings
  2. Click on the Include Tab tab.
  3. Click the Add button at the bottom.
  4. Paste http://api.unsplash.com/* into the Host field and press the Tab key on your keyboard. All off the fields will be populated here.
  5. Click OK.

Once you start adding entries here, Charles will not record traffic to other domains. This really helps in keeping your traffic logs nice and clean. It really helps when you submit defects/tickets to other people (say the web service team) with a nice clean Charles log showing what happened.

Watching Traffic

If you’ve made it this far, you’ve already seen the first major feature of Charles. You can view web traffic. This feature alone is worth purchasing a Charles license. Being able to see live data from a web service during development, testing and (even the) production phases is invaluable.

During the development phase, it’s great to see the data you are getting back in your app. Sure, you can add some debug print lines (shown above), but recompiling an app just to add logging statements you’ll need to remove later can be a pain.

Not only can you view traffic, but you can save the Charles session (File -> Save Session) for later inspection. In my day job, we have often asked QA for Charles logs along with defects that are created. This helps us know if the API (web service) is returning bad data or if the app is showing the wrong thing.

This technique of watching traffic and saving it later works for production issues too. If a single user (that you have physical access to) is having an issue, you can (temporarily) install the Charles SSL Cert and watch their traffic through Charles. This particular user may have something odd in their setup or account that’s causing something funky going on.

Please Note: This last technique where you view a customer’s data should be used sparingly. We never used it with external, paying customers. We always used this technique with internal customers. You should have the user’s permission before doing so. Having access to your customers’ logs can mean you may have access to their login information. We always have a user log into the system before we start tracking their logs. Just be careful here.

Mapping

Ok. So, I can watch traffic with Charles. What else does the app do?

One of the major benefits of using Charles is to map responses somewhere else. You can map responses to either a local file, or to a remote server. In this blog post, we’ll demonstrate mapping an API call in my Unsplashed demo app to a remote service that always returns a 500 HTTP Status code.

Using this technique is perfect for testing failures from a remote API that your app depends on without changing code to simulate the error or actually taking down the service. You can test this on an app build that is ready to ship without having to make changes. The thought of taking down a service so you can test error conditions is terrible. Don’t do that.

Remote mapping to 500 service

Great. You’re on board and want to try out this mapping thing. In the GitHub repo, there is a directory called “Charles Rules” that contains the first rule we’ll be importing into Charles. You don’t need to import this rule, you can create it yourself, but I’ve added it for simplicity.

Here’s how you import the existing rule from the GitHub Repo:

  1. Open Charles and navigate to Tools -> Map Remote…
  2. Click on the Enable Map Remote checkbox to enable it.
  3. Click the Import button at the bottom.
  4. Navigate to the Charles Rules/500HttpStatusRemote.xml file and click Open.
  5. Click OK.

Or if you’d rather create the rule yourself, follow these steps:

  1. Open Charles and navigate to Tools -> Map Remote…
  2. Click on the Enable Map Remote checkbox to enable it.
  3. Click the Add button near the bottom.
  4. Paste https://api.unsplashed.com/photos/curated/?client_id=* into the Map From Host field and press the Tab key on your keyboard. All off the fields will be populated here.
  5. Paste http://httpstat.us/500 into the Map To Host field and press the Tab key on your keyboard. All of the fields will be populated here.
  6. Click OK.
  7. Click OK.

Now, if you relaunch the app, you should be presented with an error message and see that there is now traffic being routed to http://httpstat.us/500.

The service http://httpstat.us can serve up all of the HTTP status codes including 418 I'm a teapot. You can use this technique to test all sorts of error conditions. The sample project responds the same way based on any non-200 OK response, but your app may need to respond differently based on various HTTP status codes.

When you look at the Charles traffic, you will no longer see the original URL being mapped. This also means that you’ll need to listen for the mapped URL in addition to the original one. Please read above for notes on adding additional URLs.

mapped-remotely

If you look at the response overview, you’ll notice that Charles has included a note about the response:

Mapped from remote URL: https://api.unsplash.com/photos/curated/?…

This note is important so that you know what’s happening with your messages. It’s also helpful for debugging in the next section.

Rewriting Traffic

Mapping traffic can be a really powerful tool. But Charles can do more. You can also rewrite responses from an API. Why would you want to do this? I can think of two very specific examples:

First, it’s useful when you want to return a known response on every call. This could be useful for testing with an empty result set, or testing with a funky response. Either way, you can always rewrite the response so that it includes exactly the data you’re looking for.

Second, you can handle APIs that do a bad job of error handling. I’ve worked with a few APIs that would return a 200 OK response for an error, but include JSON or XML that would include an error message. I know, we all make mistakes. Services like http://httpstat.us can’t handle this. Rewriting traffic will allow you handle any case you need to.

Rewriting

I’ve included a few example rewrite rules in the Charles Rules folder in the GitHub repo. You can import these using the following steps.

  1. Open Charles and navigate to Tools -> Rewrite…
  2. Click on the Enable Rewrite checkbox to enable it.
  3. Click the Import button at the bottom.
  4. Navigate to the Charles Rules/422RewriteError.xml file and click Open.

Creating a rewrite rule from scratch is not always easy. They can be tricky to setup and it’s easy to forget steps.

This example will return a HTTP status code of 422 (Unprocessable Entity) along with an array of error messages. This is taken directly from the Unsplash API Error Messages docs. If I were to take this example app further, I would add code to handle these error responses.

There are a lot of steps here. Please bear with me. Or watch the video below.

  1. Open Charles and navigate to Tools -> Map Remote…
  2. Click on the Enable Rewrite checkbox to enable it.
  3. Click the Add button near the bottom.
  4. Type a name into the Name field.
  5. Click the Add button under the Location section
  6. Paste https://api.unsplash.com/photos/curated/?client_id=* into the Host field and press the Tab key on your keyboard. All off the fields will be populated here.
  7. Click OK
  8. Click the Add button under the bottom Type/Rules section
  9. Change the Type dropdown value to Body
  10. Click the Request checkbox to that it is disabled
  11. Click the Response checkbox to that it is enabled
  12. Paste { "errors": ["Username is missing", "Password cannot be blank"] } into the Value field in the Replace section at the bottom.
  13. Click OK
  14. Click the Add button under the bottom Type/Rules section
  15. Change the Type dropdown value to Response Status
  16. Paste 422 Unprocessable Entity into the Value field in the Replace section at the bottom.
  17. Click OK
  18. Click Apply
  19. Click OK

rewrite

When creating these rewrite rules, it’s valuable to make sure the responses are either valid XML or JSON before using them. It can be painful to edit parts of responses in the Charles rewrite modal view. I’ll usually verify that the JSON is valid before pasting it in there. In the past, I’ve spent a lot of time trying to debug my rules only to find out the response JSON is invalid.

Conclusion

I hope this primer has been useful on getting started with Charles Proxy as a development tool. There are many more features to Charles Proxy that I haven’t even touched on.
 Charles Proxy is invaluable to me as a client developer. There are too many features to cover in this article. In fact, I’m still discovering new features that I hope to use in the future.

As I continue to grow and learn more, I may cover new topics in a subsequent post, if there’s enough interest.

This post has been updated to fix typos.

10,000 Steps a Day

A few years ago, I had a Fitbit Flex. I got slightly obsessed with getting my 10k steps in everyday. I wasn’t always successful, but it was fun (and healthy) to try.

Now, I have a Apple Watch Nike+ (Series 2). I had a series 0 Apple Watch when they first came out. I even wrote about my thoughts on that version of the watch here: How I Use Apple Watch, my thoughts on the watch have changed a bit since then, but that’s for another post.

Now I am not only trying to maintain that 10k steps/day, but I am also trying close the rings everyday. I’ve found that if I get 10,000 steps in a day, I’ll usually have completed all three rings (move, exercise and stand).

I usually go for a 1 mile walk twice a day. Usually in the morning and at lunchtime or after work. I also workout 4 - 6 times a week. A workout is usually a run, bike ride or strength training. I don’t have any issues closing the rings daily. Usually, as long as I get a walk or two in, I’ll be fine.

According to Activity++, I’ve achieved my goals for at least 30 days.

Activity++ Mid-January

At work we even have a “Step Challenge” going on. We’re seeing who (as an individual and teams) can get an average of the most steps each day in the month of January. There are really nice monetary rewards for being the first individual and on the first place team. As of today, I am in 10th place and on the third place team. My team spent most of the month in first place, but recently dropped to third.

Tracking steps and closing the rings is a fun way to try to stay healthy. I’m going to keep this going.