Downtime Window Tonight - 4/11/2014

Starting at around 2014-04-10 20:00 UTC (approximately 22 hours ago) we noticed elevated memory usage on our primary database server. While primary API services continue to operate normally, we have temporarily disabled recurring transaction processing. We are working diligently to pinpoint the problem and release memory to its normal level without the need for a database server restart. However, we are scheduling a downtime window from 2014-04-12 02:00-03:00 UTC (10pm-11pm EDT/7pm-8pm PDT tonight) to reboot our primary database server if necessary. Once the db layer is stable, recurring processing will be reenabled and delayed transactions will be queued for processing.

We expect resulting downtime to be 30 seconds or less. However, if we experience an issue requiring fail over to our backup database server, we could experience as much as 10 minutes of downtime.

This is in no way related to the “Heartbleed” bug, which CheddarGetter remains wholly unaffected by, and is separate from the cut over to our new hosting platform still scheduled for Monday morning. In fact, this scenario is the type of issue that our new high availability system will help mitigate. After Monday, we will be able to rotate database servers in and out without experiencing noticeable downtime.

If you have any questions or concerns, please don’t hesitate to contact our team via our support forum, Twitter, or Facebook.

Thanks,

Team Cheddar

Subscription Billing

Maintenance Window For April 14, 2014

After months of planning, we’re excited to make the final cut over to our new hosting platform. This new platform is engineered to provide scalability and redundancy at every level. As we continue to grow, we’re excited to enhance performance, security and reliability for all CheddarGetter customers.

As a result of the migration, we’ll be experiencing a maintenance window on Monday, April 14, between 11:00 and 12:00 UTC (7am-8am EDT/4am-5am PDT).

We expect no more than five minutes of actual downtime during the process, and we’re working diligently to keep that time to a minimum.

As we migrate to our new platform, you need to be aware of an important change that may affect your service.

If your system does any IP-based filtering or verification when communicating with CheddarGetter, you will need to ensure your system allows communication with our new IP addresses.

Our current IPs are: 67.23.3.70 and 67.23.4.246

The new cheddargetter.com IPs will be: 198.90.23.195, 198.90.23.190, 198.90.23.192, 198.90.23.194 and 198.90.21.112

Depending on your system configuration, you may have to make modifications to your system to allow communication with our new IP addresses. Customers who are using Ogone as their payment gateway or are restricting their Web Hooks by IP address should pay special attention to this. We suggest you verify your configuration with your development team and make any necessary changes before April 14.

If you have any questions or concerns, please don’t hesitate to contact our team via our support forum, Twitter, or Facebook.

Thanks for growing with us,

Team Cheddar

Subscription Billing

Scheduled Downtime For March 19th, 2014

As a preliminary step toward migration to an improved hosting platform, we’ll be experiencing a few seconds of scheduled downtime this Wednesday, between 12:00 and 13:00 UTC (7am-8am EDT/4am-5am PDT).

We’re looking forward to this initial step in our larger migration project. We’ve been working behind the scenes for months developing a new hosting infrastructure. The new platform is engineered to provide scalability and redundancy at every level. As we continue to grow, we’re excited to be able to enhance performance, security and reliability for all CheddarGetter customers.

Over the coming weeks you’ll hear more about our final migration timing. We’ve worked tirelessly to make the migration have near zero downtime. We are aiming for less than a minute of total downtime throughout the process.

If you have any questions or concerns, please don’t hesitate to contact our team via our support forum, Twitter, or Facebook.

Thanks for growing with us,

Team Cheddar

Subscription Billing

Trello - Home of the CheeseMap

By now you’ve all probably heard of the awesome project management site Trello. Which means that you should be more than familiar with The CheeseMap - our CheddarGetter development board.

  • Want to know what we have in the works?
  • Have input on one of our suggested features?
  • Want to suggest some features of your own?
You can communicate with our Dev Team, vote for the features that matter to you, and much more just by keeping your eyes on The CheeseMap.
      Subscription Billing

      Need Some Help?


      Never fear, because now it’s easier than ever to get CheddarGetter support - or just to reach out and say hi - on Twitter, Tumblr, Facebook, or Google+.

      If we were any more heroic we’d need capes.

      Subscription Billing

      Zapier Is Giving You Even More App Integration Possibilities

      We’ve talked about how fantastic Zapier is in the past (*Spoilers* You can use Zapier to integrate CheddarGetter with a phenomenal number of services), so as you can imagine it makes us pretty pumped to announce that those crazy kids are at it again.

      In honor of “the month of love”, they’re adding 28 new apps to Zapier over the next 28 days of February. That means even more CheddarGetter combination possibilities.

      You can read the full announcement on their blog here - including a peek at the first batch of three new Zapier-ready apps - and tune in daily for new integration announcements.

      Subscription Billing

      A Look Back, A Look Forward, And A Flying Cat

      Time Flies. Which is why we’re doing a look back on everything that we accomplished in 2013 on 1/31/14…but it’s never to late to give an update on what you may have missed, or a forecast of all of the awesome things we have in store for 2014!

        Here’s a rundown on what we accomplished in 2013:

        • Native Zapier Support - Zapier could be the single most awe-inspiring addition to CheddarGetter in 2013. You can pretty much do anything. Check it out.
        • Significant improvements to customer lookup (speed, ease of use, etc).
        • Improved processor communication to increase fault tolerance and better handling for general internet glitches
        • Various improvements to PayPal Preapproval workflow (more to come!)
        • Massive overhaul of database schema including primary key switcheroo on several key tables with zero downtime!
        • Billing address info added to email template variables
        • Enable customers to enter coupon codes in hosted pages
        • Limit promotion application to certain charge types
        • Additional billing solution support
        • Increased PayPal preapproval limits to $10000 and 3 years
        • Resend invoice email receipts via API and GUI
        • Improvements to duplicate POST check
        • Refactored hook system. It’s not even more powerful and reliable. Added Bill Reminder as a hook event.
        • Improvements to credit card validation when payment method update does not result in a revenue transaction
        • Additional config options for Bill Reminder events.
        • Found and fixed some memory leaks in our background processing
        • Several additions to CG’s real-time automated auditor

        Now a sneak peek into what we’ve been working on behind the scenes:

        • Massive overhaul hosting infrastructure and migration to new hosting provider. Flipping the switch in early 2014!
        • Full rebrand and rewrite of marketing site. Easier to use documentation. Coming in Q1 2014.
        • New PCI DSS Service Provider Level 1 assessment. Certification expected to be complete in Q1 2014.
        • Admin GUI rewrite including significant improvements to dashboard reporting and customer search
        • JSON API interface. Read-only metrics in JSON format.
        Stay tuned for more developments, features, and nature/gravity defying animals from CheddarGetter in 2014! Subscription Billing

        Balancing Usage Levels and Charges

        CheddarGetter has long been a proponent of bootstrapped startups. In every single planning meeting, we talk about our core customer base as being the startup. This has caused some degree of trouble for us when talking to larger customers, as they don’t understand how a company like ours can grow and sustain itself by catering to what they see as a fickle, unprofessional group. In our experience, professionalism, talent, and potential have little or nothing to do with a company’s age. It has everything to do with a team’s integrity and standards.

        In our search for balance between catering to startups with a low-cost offering, and servicing larger businesses with a powerful and flexible offering, we have made some pricing decisions that have negatively affected our ability to service either.

        We currently charge our merchants for any transaction that results in a movement of money from the end customer to the merchant (you, perhaps). That works great if all end customers pay the merchant monthly. In that case, we collect $0.20/customer/month and that covers all of the expenses we incur to service that account over that month plus a little. Unfortunately in many cases, that’s not what happens.

        Our current model does not accommodate for other hard costs or high levels of non-revenue generating activity. In other words, other types of services including web hooks and API calls cost money and our current model doesn’t cover for those costs when usage of those services exceeds the norm.

        As such, we’re writing this post to start a discussion about pricing for a few valuable services.

        API Calls

        In most cases, API calls to CheddarGetter occur fairly infrequently. Businesses who correctly integrate with our API, and use it as it was meant to be used, present a reasonable load on our server, and cause no problems. The cost of providing an API is just a cost of doing business, and we have no desire to charge any more than what is necessary to cover those costs. Most will never pay us directly for a single API call but in the case where a merchant’s usage of the API is significantly disproportionate to the amount of revenue we receive from that customer, we need to do something about it.

        With API calls, there may be an additional charge for exceeding a set velocity or exceeding a set total number of API calls in a month. In each case, the limits will be set relative to, say, the number of revenue transactions in a month. We’re not yet doing this but plan to soon. First, we will begin tracking API call quantities and velocities to get a baseline measurement. Based on that data, we will determine what’s normal, then decide at what level to start charging. Our goal here is only to charge for usage well above the average so the vast majority of our customers will not be charged for API calls. Only those with an abnormally high usage will be charged. The fees, if any, will be nominal.

        Web Hooks

        As with API calls, CheddarGetter handles your web hooks with ease in most cases. However when your services hits a snag (something hangs), our service retries that hook multiple times, for multiple seconds each time. This, of course, is a good thing because it makes the hook system tolerant of failure of the listener to process the hook. This can cause significantly elevated usage of resources by our background processes, which can delay other customers’ webhooks if there’s a backlog. This is bad juju, and we feel that good users should not bear the weight of those who are causing the problems.

        Similar to API calls, we’re looking at setting some velocity and total limits as well as total time waited for listeners to respond. We’re not doing this yet but plan to soon. Again, only those with abnormally high usage of hooks will be affected.

        Transactions

        As you know, transaction charges are a part of life. Some processors charge percentage fees, some charge flat rates, others charge both. We’ve always felt that a flat fee is the most fair method to cover these costs, because it puts a standard value on the process of transacting, rather than sliding number up or down to capture a larger piece of your higher-dollar transactions. In real terms, paying 3% of $100 is a lot harder to swallow than paying $0.20, compared to the difference between paying $0.20 on a $10 transaction vs $0.03. This varies of course depending on your pricing model, but in the end we determined that it was the most equitable model for our customer base.

        Transactions are a hard cost for us as well. In the past, we have charged only for Approved (Successful) and Refunded transactions. These are $0.20 for the Blowing Up plan, and $0.25 for the Paypal Only plan. What you haven’t seen are the costs for other transaction types, like Declined, Failed, Authorization (Auth), and Voided. Approved and Refunded transactions are pretty straightforward but the others are not. I’ll explain.

        Declined transactions may seem innocuous but these amount to a significant cost to for us. They occur more often than you might think. First, consider repeated failed signups, some are even attempts to fraudulently use one or more stolen cards. Second, consider dunning retries. CG by default continues to attempt to charge the customer 4 more times over 8 days before giving up. You can configure your dunning retry schedule here: https://cheddargetter.com/admin/configuration

        Failed is a special transaction that occurs when a transaction is first approved, then later found to be declined. This is typical of ACH (aka echeck) transactions. This mostly occurs when a PayPal payment is backed by a bank draft instead of a credit card. The transaction is initially approved but PayPal then later informs us that the money could not be taken from the PayPal account holder’s bank account.

        Authorizations and Voids apply only to those merchants using a validation transaction on signup or payment method change events. When a validation transaction is configured, CG issues a small authorization transaction which tests the card all the way to the issuer including CVV verification (if applicable) and AVS checks (if applicable). This is extremely valuable for those who accept payment methods with a delayed initial bill and for payment method changes mid billing cycle. Without them, you have to trust the customer to enter all of their information properly without validation other than basic format checks. If the payment information is wrong at bill time, it will be declined. That’s a hassle for everyone.

        Voids are related to the Authorization. When an Authorization is successful, we need to issue a Void to indicate that we never intend to capture that small Authorization. If this isn’t done, the Authorization will remain on the cardholder’s card statement for several weeks. Hanging Authorizations are a no-no for that reason but also because the banks consider them to be a liability when there’s a large quantity of hanging auths.

        In order to continue providing a great service, we felt that it was necessary and fair to pass these costs to the people who are using them.

        For most of our customers, these costs will amount to a very small increase on their monthly bill.

        For CheddarGetter, the sum of these costs across our entire customer base accounts for a very significant expense each month. That is money that we could and should be using to shore up our service, provide higher availability, and generally make our system stronger and better.

        We know that this is a touchy subject.

        We know that people don’t like to pay for something that they didn’t realize they were getting for free.

        We also know that our competitors already charge for these things, and it is possible that some of our customers made the decision to use CheddarGetter because we did not.

        So…

        To alleviate potential future discomfort for all involved, here is our plan.

        As of yesterday, we turned on the charge system for these “other” transactions (Auths, Voids, Declines, and Failures) on the PayPal Only and Blowing Up plans (legacy plans are not affected). Everyone is able to see these pending transaction charges, if any, on the Billing page: https://cheddargetter.com/admin/billing.

        As promised, we are giving all of you the option to tell us you don’t want to pay for them. If you are on a super-tight budget and simply cannot afford the extra charges, we understand. Or maybe you don’t think that these charges should be passed on to customers on principle. Either way, let us know and we will adjust your account so you will not be charged for those transactions.

        In Summary

        API Calls Now: No change
        Soon: Plan to charge for extreme usage / Costs TBD but will be minimal
        Take action: This topic is open for discussion here (http://support.cheddargetter.com/discussions/questions/5850-api-and-webhooks-overage-pricing).

        Web Hooks Now: No change
        Soon: Plan to charge for extreme usage / Costs TBD but will be minimal
        Take action: This topic is open for discussion here (http://support.cheddargetter.com/discussions/questions/5850-api-and-webhooks-overage-pricing).

        Transactions Now: Charge for all transaction types on current plans (legacy plans, Basic, Advanced, Premium, etc, are not affected)
        Soon: No further changes
        Take action: Review your charges as they accrue here: https://cheddargetter.com/admin/billing. Evaluate how they affect your budget in relation to the benefits to the health of CheddarGetter’s system. If acceptable, no action needed. If you choose to opt out, let us know here (team@cheddargetter.com).

        Subscription Billing

        Database Craziness. Downtime, too. Not Directly Related.

        We’ve had two episodes of lengthy downtime in the past 60 days. Early this morning we were down for almost 3 hours. Back in February it was even longer at over 4 hours on the night of the 6th. Let me first be very clear: Any downtime at all is unacceptable, planned or otherwise. Our job is to do our best to prevent this sort of thing as much as is possible. We’re currently failing in that effort.

        Both of these episodes were avoidable. No, it wasn’t a “perfect storm”. Port 443 was hung and we were asleep. In both instances, we were back up within a few minutes of cracking an eyelid.

        No, we don’t have our humans monitoring services 24/7. We should. No, we don’t have enough redundancy to avoid this particular issue. We should. Why don’t we? The answer is simple and not one that anyone wants to hear. A choice has been made to keep our hosting infrastructure in it’s current state, a vulnerable state, until we secure additional capital to improve it. We should have more humans and more redundancy. Both cost money and lots of it.

        You’re probably wondering what it is we *are* doing. I’ll give you an example. Yesterday I posted a long overdue change log of what the development team has been doing over the past few months. In one of those many bullet points, I mentioned that we had flipped our database keys - primary keys and foreign keys, from a 128 byte UUID to a basic integer. One of our favorite customers emailed me and said:

        Impressive! Any quick tips you can share for low impact DB maintenance?

        It’s quite an unsettling feeling that we are now boasting about DB maintenance fu when we just had a ridiculous amount of downtime. Well, this is actually a great example of what we have been doing to improve and how we sacrifice our time in extra effort to avoid any time offline. It’s interesting, too, and a subject of a blog post that has never surfaced. So, I’ll just describe briefly what we did and how we did it.

        Before I get started, I should explain that the reason we were forced to make this drastic change to our database schema was another fault of our own. When I first created the schema for CheddarGetter’s primary database, I decided to give UUIDs a whirl. I used them for primary keys and foreign keys. The columns are char(36). They’re hex UUIDs with dashes — the kind you get from MySQL’s UUID() function. That’s 128 bytes UTF-8. Yeah, UTF-8, entirely unecessary for these columns. Integers are far more efficient in BTREE indexes than are 128 byte strings and far smaller, too. The difference in efficiency was causing severe headaches for everyone — our dev and support teams and for our customers. We absolutely had to make this change.

        The size of our database at the time was around 12G. That’s not huge but it’s large enough that making the changes we needed to make would cause hours of planned downtime if we were to use traditional means. Since outages are unacceptable, even those for planned maintence, we figured out a way to do it without affecting live services at all, not even for a minute. Instead of a few days of work, it took more than the entire month of January. We put in the extra time in order to prevent hours, likely >10 hours of database locks. Here’s how we did it:

        1. Plan the new schema
          • New id columns
          • New primary keys
          • New foreign keys
        2. Override database access in the app framework so the app is tolerant of changes
          • Make inserts create both keys
          • Make updates reconcile both keys
          • Make lookups by primary key work for either key
        3. Make the changes

        Normally when you make schema changes, you simply use an ALTER statement. The trouble with ALTERs is that the table is locked while the alter takes place. When an ALTER is executed, the entire table must be recreated. For a single 4G table, for example, that can take a long time, maybe an hour or more. Ours was a unique situation because we were going to have to run an ALTER more than once for the same table because we were dealing with primary and foreign keys. In short, these changes cascade into other tables and had to be well orchestrated. Each table had to be altered, then each related table had to be altered, then the original table altered again, and the related tables again, too. That’s craziness, so we found a way to do it without any downtime.

        Enter Percona Toolkit's pt-online-schema-change, a well-named tool. An ALTER using pt-online-schema-change 1) creates a copy of the target table schema, 2) runs the ALTER on the new empty table, 3) copies all the rows from the old table, 4) ensures the two tables are in sync, 5) replaces the old table with the new. There’s a lot of database trigger witchcraft mixed in there and it works quite well. The sacrifice is time. It takes around 10x longer to alter a table using pt-online-schema-change than a traditional alter.

        Here’s a simple example of what we needed to do and how we did it with pt-online-schema-change:

        Consider this simple schema:

        CREATE TABLE Caves (
        	id char(36) not null primary key,
        	location varchar(255),
        	createdDatetime datetime
        ) ENGINE=InnoDB;
        
        CREATE TABLE Bears (
        	id char(36) not null primary key,
        	caveId char(36) not null,
        	constraint caveId_fk foreign key (caveId) 
        		references Caves(id),
        	name varchar(255),
        	createdDatetime datetime
        ) ENGINE=InnoDB;	
        

        First, alter the table remove the primary key, add the new primary and set an index on the old primary:

        alter table Caves 
        	add unique index id(id), 
        	drop primary key, 
        	add column newId int unsigned 
        		not null primary key auto_increment FIRST;
        

        … and the pt-online-schema-change equivalent (two steps required):

        pt-online-schema-change --alter \
        "add column newId int unsigned not null \
        auto_increment FIRST, add index newId(newId)" \
        D=testdb,t=Caves --lock-wait-time=50 \
        --alter-foreign-keys-method auto \
        --nocheck-replication-filters --dry-run
        
        pt-online-schema-change --alter \
        "drop primary key, drop index newId, \
        add primary key(newId), add unique index id(id)" \
        D=testdb,t=Caves --lock-wait-time=50 \
        --alter-foreign-keys-method auto \
        --nocheck-replication-filters --dry-run
        

        Next, do the same for the related table(s) but also make foreign key change(s):

        alter table Bears 
        	add unique index id(id), 
        	drop primary key, 
        	add column newId int unsigned 
        		not null primary key auto_increment FIRST,
        	add column caveNewId int unsigned not null after id;
        
        pt-online-schema-change --alter \
        "add column newId int unsigned not null \
        auto_increment FIRST, add index newId(newId)" \
        D=testdb,t=Bears --lock-wait-time=50 \
        --alter-foreign-keys-method auto \
        --nocheck-replication-filters --dry-run
        
        pt-online-schema-change --alter \
        "drop primary key, drop index newId, \
        add primary key(newId), add unique index id(id), \
        add column caveNewId int unsigned after id" \
        D=testdb,t=Bears --lock-wait-time=50 \
        --alter-foreign-keys-method auto \
        --nocheck-replication-filters --dry-run
        

        Populate the new foreign key:

        update Bears b, Caves c 
        	set b.caveNewId=c.newId 
        	where b.caveId=c.id;
        

        We did this a block at a time using the `watch` command:

        watch -n 0.1 'mysql -e "select count(*) from Bears \
        where caveNewId is null; update Bears b \
        set b.newCaveId=(select b.newId from Bears b \
        where b.caveId=c.id) \
        where b.newCaveId is null limit 50000;" testdb'
        

        Last, setup the new foreign key(s):

        alter table Bears 
        	add constraint cave_fk foreign key (caveNewid) \
        	references Caves(newId);
        
        pt-online-schema-change --alter \
        "cave_fk foreign key (caveNewid) \
        references Caves(newId)" D=testdb,t=Bears \
        --lock-wait-time=50 --alter-foreign-keys-method auto \
        --nocheck-replication-filters --dry-run
        

        For one of our tables, that process took 6 days.

        Alright, long story short, we’re working hard to improve and to avoid as much downtime as possible. Things are going to get better. Honest.

        Subscription Billing

        Integrate CheddarGetter With Anything, Using Zapier!

        Zapier is a sort of universal adapter for APIs. We have worked with their team to integrate CheddarGetter’s API in the most comprehensive way possible. This means that you can use Zapier to connect your CheddarGetter account to a ton of other services across the web.

        If you want to try out some “zaps”, sign up for an account using our coupon code and you can get a 50% discount if you decide to upgrade to their paid plan. It’s free to start.

        → Try Zapier using code: cheddargetter_ef223


        Zapier Integration Recipes

        Here are a few example recipes using some of the services we regularly use at CheddarGetter:

        1. New Subscription to SMS
        2. New Subscription to MailChimp
        3. New Subscription Email Draft
        4. New Subscription to Basecamp Message
        5. New Subscription to Highrise Contact
        6. New Transaction to Ambassador

        If you don’t see what you need, you can make your own “Zaps” very easily with their simple editor.

        Try some of these combinations:

        1. CheddarGetter with Freshbooks
        2. CheddarGetter with Quickbooks
        3. CheddarGetter with Zendesk
        4. CheddarGetter with Magento
        5. CheddarGetter with Salesforce

        Zapier Is Open

        The best thing about integrating with Zapier? It’s completely open. This means that any service you happen to use is a potential integration for CheddarGetter. You can scan their giant list of companies that are already integrated with Zapier, and if you find something you like, create a recipe between CheddarGetter and the system of your choosing.

        See the list of currently possible integrations here 

        Open also means that if your service is not on that list, you can add it yourself. That requires a bit of coding on your part, but if it’s something you think is worthwhile, chances are someone else will too. Once you write an integration, it’s available for all other Zapier users. Good karma all around.

        Subscription Billing