Posts - Scott Robinson · @ssxio

How to Detect Service Versions on Adobe Commerce Cloud

2026-04-10T00:00:00-00:00

Adobe Commerce Cloud manages your infrastructure services for you, but knowing exactly which versions of Valkey, MariaDB and RabbitMQ are running on your environments isn’t always obvious. The .magento/services.yaml file defines what you’ve requested, but that doesn’t always reflect what’s actually deployed - especially after platform upgrades or environment redeployments.

Here’s how to check the actual running versions by SSH’ing into your environment.

Valkey

$ valkey-server -v
Valkey server v=8.0.1 sha=4fbab574:0 malloc=jemalloc-5.3.0 bits=64 build=84a9272b225dd059

If your environment is still running Redis rather than Valkey, you can use redis-server -v instead.

MariaDB

$ mariadb -V
mariadb  Ver 15.1 Distrib 10.11.9-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

Note the uppercase -V flag here - lowercase -v won’t give you what you need.

RabbitMQ

$ dpkg -s rabbitmq-server | grep Version
Version: 4.1.0-1

RabbitMQ doesn’t have a straightforward version flag like the others, so querying the installed package via dpkg is the simplest approach.

Why this matters

Knowing your actual service versions is useful when debugging compatibility issues, preparing for upgrades or verifying that a platform change has actually taken effect. It’s also handy when raising support tickets with Adobe - having the exact version numbers saves a fair amount of back and forth.

Blocking SessionReaper and PolyShell File Access with Fastly on Magento/Adobe Commerce

2026-03-30T00:00:00-00:00

If you’ve been keeping an eye on the Magento security space recently, you’ll have seen the research from Sansec around SessionReaper exploitation and the more recent PolyShell - both of which use the media/custom_options/ directory as a vector for uploading and executing malicious PHP files.

The SessionReaper attack targets session data to extract customer and admin credentials, and the PolyShell technique builds on this by disguising executable PHP within files that appear to be valid image uploads.

Blocking access at the edge with Fastly

If you’re running Adobe Commerce Cloud or any Magento instance fronted by Fastly, you can block access to these paths before requests ever reach your origin server. This is a fairly simple custom VCL snippet that returns a 403 for any request targeting media/custom_options/:

if (req.url ~ "^/media/custom_options/") {
  error 403 "Forbidden";
}

Add this as a custom VCL snippet in your Fastly configuration with the type set to recv and priority 1. This will catch any direct access attempts to files within that directory - whether they’re SessionReaper scripts, PolyShell payloads or anything else that has no business being served publicly.

Edge-level blocking over application-level blocking

Blocking at Fastly rather than at the application level means the request never reaches nginx, Varnish or PHP. This is important for two reasons - it reduces the attack surface significantly and it means even if an attacker has managed to upload a malicious file, they can’t execute it by requesting it directly through the public URL.

You should still be auditing your media/custom_options/ directory for any suspicious files and ensuring your Magento instance is patched, but adding this edge-level block is a solid defensive layer that takes seconds to deploy.

Important Note

If your store actively uses custom options with file upload fields on products, you’ll need to be more careful with this approach. Blanket blocking the entire media/custom_options/ path could interfere with legitimate file uploads that customers need to access. In that case you may need to refine the VCL to target specific file extensions or patterns rather than the whole directory.

Fixing Magento's Telephone Validation to Allow Dots and Slashes

2026-03-23T00:00:00-00:00

Magento’s built-in telephone validation for customer accounts is fairly strict - it only allows digits, spaces, parentheses, plus signs and hyphens. That works for a lot of western formats but breaks for plenty of valid international phone numbers.

French phone numbers are commonly written with dots as separators (e.g. 06.76.40.32.22), and Austrian numbers often use a forward slash (e.g. +43680/2149568). Both of these formats are rejected by Magento’s Telephone validator out of the box.

The Telephone model validator in vendor/magento/module-customer/Model/Validator/Telephone.php uses a regex pattern that doesn’t account for dots or slashes:

private const PATTERN_TELEPHONE = '/(?:[\d\s\+\-\()]{1,20})/u';

This means any customer trying to save their phone number in a perfectly valid international format gets hit with “Invalid Phone Number” - not a great experience.

The issue is tracked internally as ACP2E-4554 and was discussed in magento/magento2#40544 and magento/magento2#40545. The fix was committed but if you’re on a version that doesn’t include it yet, here’s a patch.

Here’s an m2-hotfixes patch that adds support for dots and forward slashes in telephone validation:

--- a/vendor/magento/module-customer/Model/Validator/Telephone.php
+++ b/vendor/magento/module-customer/Model/Validator/Telephone.php
@@ -21,9 +21,11 @@
      * \() :Matches open and close parentheses
      * \+: Matches the plus sign.
      * \-: Matches the hyphen.
+     * \.: Matches the dot character (e.g. 06.76.40.32.22)
+     * \/: Matches the forward slash (e.g. +43680/2149568)
      * \d: Digits (0-9).
      */
-    private const PATTERN_TELEPHONE = '/(?:[\d\s\+\-\()]{1,20})/u';
+    private const PATTERN_TELEPHONE = '/(?:[\d\s+().\/ -]{1,20})/u';

     /**
      * Validate telephone fields.
@@ -35,7 +37,7 @@
     {
         if (!$this->isValidTelephone($customer->getTelephone())) {
             parent::_addMessages([[
-                'telephone' => "Invalid Phone Number. Please use 0-9, +, -, (, ) and space."
+                'telephone' => "Invalid Phone Number. Please use 0-9, +, -, (, ), ., / and space."
             ]]);
         }

Save this as m2-hotfixes/ac-13791__telephone-validation-dots-slashes.patch (or whatever naming convention you use for your hotfixes directory).

The updated regex pattern '/(?:[\d\s+().\/ -]{1,20})/u' adds . and / to the allowed character set.

Creating URL Redirects When Updating Product URL Keys via the Magento API

2026-03-12T00:00:00-00:00

If you’ve ever updated a product’s URL key via the Magento 2 REST API and then wondered why the old URL is now returning a 404, you’ve probably missed the url_key_create_redirect custom attribute.

In the admin UI there’s a handy “Create Permanent Redirect for old URL” checkbox that appears when you change a product’s URL key. Behind the scenes, this sets url_key_create_redirect on the product - and you can pass exactly the same attribute in your API requests to get the same behaviour.

The API request

When updating a product’s URL key, include url_key_create_redirect alongside url_key in your custom_attributes array:

PUT /rest/V1/products/{sku}

{
  "product": {
    "custom_attributes": [
      {
        "attribute_code": "url_key",
        "value": "new-product-url-key"
      },
      {
        "attribute_code": "url_key_create_redirect",
        "value": "1"
      }
    ]
  }
}

Setting the value to 1 tells Magento to create a 301 redirect from the old URL to the new one. The redirect is written to the url_rewrite table with redirect_type set to 301.

If you want to suppress redirect creation entirely - for example during a bulk initial import where no old URLs exist yet - set the value to 0 or omit the attribute.

Why this matters for bulk operations

If you’re running a bulk URL key update across hundreds or thousands of products, you’ll want to be deliberate about this. Passing url_key_create_redirect: 1 on every product creates a redirect entry for each one, which can pile up in url_rewrite over time - especially if products have had multiple URL key changes.

It’s worth cleaning out stale redirects periodically, or being selective about which updates warrant a redirect versus which are clean-slate changes (new products, dev/staging refreshes, etc.).

Verifying the redirect was created

You can check the url_rewrite table directly after the API call:

SELECT *
FROM url_rewrite
WHERE entity_type = 'product'
  AND redirect_type = 301
  AND target_path LIKE '%new-product-url-key%'
ORDER BY url_rewrite_id DESC
LIMIT 10;

Or use the REST API to inspect the product’s URL rewrites:

GET /rest/V1/url-rewrites/products/{product_id}

With a bit of luck this saves you the debugging time of wondering why your URL key updates are silently dropping old links.

Fixing Broken Product Relationships in Adobe Commerce/Magento 2

2026-02-17T00:00:00-00:00

I recently ran into a fairly puzzling issue on an Adobe Commerce site where some products were behaving oddly on the frontend. After some digging, it turned out the catalog_product_relation and catalog_product_super_link tables had invalid entries - simple products were listed as parents with child relationships, which should never happen.

In Magento/Adobe Commerce, only configurable, bundle and grouped products should have child relationships. When simple products end up with children in these tables, it’s usually the result of a botched import, a failed product type conversion or some other data corruption.

Finding the problem

First, identify any invalid entries in catalog_product_relation where the parent is a simple product:

SELECT
    cpr.parent_id,
    cpr.child_id,
    parent.entity_id as parent_entity_id,
    parent.sku as parent_sku,
    parent.type_id as parent_type,
    child.entity_id as child_entity_id,
    child.sku as child_sku,
    child.type_id as child_type
FROM
    catalog_product_relation cpr
INNER JOIN
    catalog_product_entity parent ON cpr.parent_id = parent.row_id
INNER JOIN
    catalog_product_entity child ON cpr.child_id = child.row_id
WHERE
    parent.type_id = 'simple'
ORDER BY
    parent.entity_id, child.entity_id;

Similarly, check catalog_product_super_link for non-configurable parents (this table should only contain configurable product relationships):

SELECT
    cpsl.link_id,
    cpsl.product_id,
    cpsl.parent_id,
    parent.entity_id as parent_entity_id,
    parent.sku as parent_sku,
    parent.type_id as parent_type,
    child.entity_id as child_entity_id,
    child.sku as child_sku,
    child.type_id as child_type
FROM
    catalog_product_super_link cpsl
INNER JOIN
    catalog_product_entity parent ON cpsl.parent_id = parent.row_id
INNER JOIN
    catalog_product_entity child ON cpsl.product_id = child.row_id
WHERE
    parent.type_id != 'configurable'
ORDER BY
    parent.entity_id, child.entity_id;

You can also get a quick count of how many invalid entries exist:

SELECT 'catalog_product_relation' as table_name,
       COUNT(*) as invalid_count,
       'Parents with type_id=simple' as issue
FROM catalog_product_relation cpr
INNER JOIN catalog_product_entity parent ON cpr.parent_id = parent.row_id
WHERE parent.type_id = 'simple'

UNION ALL

SELECT 'catalog_product_super_link' as table_name,
       COUNT(*) as invalid_count,
       'Parents with type_id!=configurable' as issue
FROM catalog_product_super_link cpsl
INNER JOIN catalog_product_entity parent ON cpsl.parent_id = parent.row_id
WHERE parent.type_id != 'configurable';

Checking for orphans

While you’re at it, check for orphaned references where the parent or child product no longer exists:

-- Missing parents in catalog_product_relation
SELECT cpr.parent_id, cpr.child_id, 'Missing parent' as issue
FROM catalog_product_relation cpr
LEFT JOIN catalog_product_entity parent ON cpr.parent_id = parent.row_id
WHERE parent.row_id IS NULL;

-- Missing children in catalog_product_relation
SELECT cpr.parent_id, cpr.child_id, 'Missing child' as issue
FROM catalog_product_relation cpr
LEFT JOIN catalog_product_entity child ON cpr.child_id = child.row_id
WHERE child.row_id IS NULL;

Cleaning up

Back up your database first. Then remove the invalid entries:

-- Remove invalid entries from catalog_product_relation
DELETE cpr
FROM catalog_product_relation cpr
INNER JOIN catalog_product_entity parent ON cpr.parent_id = parent.row_id
WHERE parent.type_id = 'simple';

-- Remove invalid entries from catalog_product_super_link
DELETE cpsl
FROM catalog_product_super_link cpsl
INNER JOIN catalog_product_entity parent ON cpsl.parent_id = parent.row_id
WHERE parent.type_id != 'configurable';

After cleanup, reindex and clear cache:

bin/magento indexer:reindex
bin/magento cache:clean

One thing to consider - if those simple products should actually be configurable products, you’ll need to convert them rather than just deleting the relationships. The queries above assume the relationships themselves are the problem, not the product types.

Disable Chrome AI Mode in the Omnibox

2026-02-09T00:00:00-00:00

If you want to disable the AI mode button that appears in Chrome’s omnibox/address bar then you can do so via Chrome flags.

Navigate to:

chrome://flags/#ai-mode-omnibox-entry-point

Set the flag to Disabled and relaunch Chrome.

Claude Code - How to Revert Back to Opus 4.5 from Opus 4.6

2026-02-02T00:00:00-00:00

If you want to go back to Opus 4.5, start hashtag#claude with:

claude --model claude-opus-4-5-20251101

You can also edit your config file to set this as the default if you want to use it by default rather than 4.6.

Signing Git Commits with SSH

2026-01-26T00:00:00-00:00

I recently ran into a “No signature” error when trying to get my git commits verified and it sent me down a bit of a rabbit hole. If you’ve ever tried to set up GPG signing for git commits, you’ll know it can be a fairly painful experience.

As you can now sign git commits with SSH keys rather than GPG and given that pretty much everyone already has an SSH key (and pretty much no one has a GPG key properly configured) - this is a much more practical approach.

Caleb Hearth has written an excellent post on how to set this up.

If you’re using GitHub, you’ll need to upload your SSH key separately as a signing key (distinct from your authentication key) - but once that’s done you’ll get the verified badge on your commits.

Adobe Commerce Isolated Security Patches Changelog

2026-01-12T00:00:00-00:00

Adobe have changed the way they usually release security patch updates for
2026, but sadly their communcations around the release leaves quite a bit
to be desired.

I’ll collate the release information here through the year and document
what has/hasn’t been released as per the official release schedule.

13/01/2026: No patches released, confirmed by Nathan Smith via Slack.

10/02/2026: There was a release planned but then was revoked. Currently no release planned.

10/03/2026: Pending

14/04/2026: Pending

12/05/2026: Pending, should be full semver release (well, Adobe’s attempt at it).

09/06/2026: Pending

14/07/2026: Pending

11/08/2026: Pending

08/09/2026: Pending

13/10/2026: Pending

10/11/2026: Pending

08/12/2026: Pending

Adobe Commerce Deployments Failing - Composer block-insecure in Effect for 2.7/2.8

2026-01-02T00:00:00-00:00

If you’re running into this issue with Adobe Commerce deployments, it’s due to the older versions of composer now being marked as having a few security issues (“PKSA-1gck-s111-yq7g”, “PKSA-b8f7-zn44-r4gz”, “PKSA-s25b-vbmp-jvhh”):

Installing build dependencies...
  Installing php build dependencies: composer/composer
  W: Changed current directory to /app/.global/composer/composer
  W: No composer.lock file present. Updating dependencies to latest instead of installing from lock file. See https://getcomposer.org/install for more information.
  W: Loading composer repositories with package information
  W: Updating dependencies
  W: Your requirements could not be resolved to an installable set of packages.
  W:
  W:   Problem 1
  W:     - Root composer.json requires composer/composer 2.7.0 (exact version match: 2.7.0 or 2.7.0.0), found composer/composer[2.7.0] but these were not loaded, because they are affected by security advisories. To ignore the advisories, add ("PKSA-1gck-s111-yq7g", "PKSA-b8f7-zn44-r4gz", "PKSA-s25b-vbmp-jvhh") to the audit "ignore" config. To turn the feature off entirely, you can set "block-insecure" to false in your "audit" config.
  W:

E: Error building project: `composer` could not be run.

E: Error: Unable to build application, aborting.

To fix the issue, update the version of composer used to build your project in the .magento.app.yaml file:

dependencies:
    php:
        composer/composer: '^2.9'

For Magento 2.4.7 and up, use ^2.9, for anything else use ^2.2 which should give you 2.2.26.

Nginx client_header_buffer_size - Debugging Cloudflare-related 400 Errors for Magento/Adobe Commerce HTTP Requests

2025-12-24T00:00:00-00:00

One of the more interesting things I’ve had to debug this year was why Magento
was apparently returning 400 responses to builder.io for certain webhook payloads.

A few of the team had looked into this but drawn a bit of a blank so I offered to take a look and it ended up being a fairly puzzling one to debug.

The same payload POST direct to a local running version of the application would give the expected 200 response, but when running through it’s production stack of Cloudflare and into the excellent Corefinity - we suddently started getting 400 response codes which was fairly odd as the custom controller that was receiving the payload had no code in it to return a 400 response.

My first thought was that Cloudflare was stopping the requests going due to either the body content or an OWASP ruleset. Running the requests through in no-proxy mode discounted this as being the issue.

A fairly important piece of debugging spotted that the nginx log presented the 400 error as well, so that ruled out Cloudflare being the issue.

I added some ancillary logging to fire whenever the controller Magento side fired to determine whether or not the request was even getting that far and fairly quickly realised that they weren’t even hitting Magento to return a 400 response.

My thoughts then shifted to either PHP’s post_max_size, or perhaps
max_input_vars set to low and the payload couldn’t be parsed. The additional logging never fired when the 400 was being returned so that pointed to nginx or varnish being the culprit.

At this point, I trapped a few of the outbound POST requests again and spotted that once they’d been through Cloudflare there was quite a number of headers and suspected that the issue we were actually running into was that nginx’s two configuration values were too low for:

client_header_buffer_size
large_client_header_buffers

This turned out to be the case, once these were increased the 400 errors soon
disappeared. We did also debug whether varnish was involved but fairly quickly
ruled out that being part of the issue. It was solely nginx.

If you ever encounter add odd 400 response for a webhook on any host, I can highly recommend that you check those two configuration values first and with a bit of luck I’ve saved you some debugging.

The best piece of advice I can give you is to isolate the point inside the stack you’re getting to when the 400 is thrown, that will help you narrow down what the cause is quickly.

Adobe Commerce/Magento 2 Concurrent Import Process Interference Issue

2025-12-04T00:00:00-00:00

If you’re running multiple import processes simultaneously in Adobe Commerce or Magento 2, you might encounter a frustrating bug where the imports interfere with each other, causing data corruption or import failures. This issue, documented in GitHub issue #37593, has been confirmed in Magento 2.4.6 but is present in other versions too namely 2.4.7.

If two import processes run concurrently in Magento 2.4.6, they can interfere with each other in unexpected ways. For example, if you’re simultaneously importing customers and products, product data might erroneously appear in the customer import process, causing failures. I discovered this while debugging a customer importer that ended up being fed stock source import data.

The issue stems from how Magento handles large import files using the importexport_importdata table. The default batch size limit is set to 100 records in vendor/magento/module-import-export/etc/config.xml. When an import exceeds this threshold, Magento splits the data across multiple rows in the database table and this is fed into the import
engine.

The problem occurs when concurrent imports add their validated data simultaneously to this shared table. Processes can then retrieve mixed or incorrect data from other running imports, leading to validation errors and data corruption.

Reproduction

- Open Magento admin in two separate browser sessions or systems
- Navigate to System > Data Transfer > Import in both
- In the first session: Select "Customers and Addresses" with 500+ records
- In the second session: Select "Catalog Product Import" with 50-60 records
- Validate both imports
- Click Import on the first system, then immediately click Import on the second

It’s fixed in this commit which was pushed out in 2.4.8. For 2.4.7, Adobe have provided this ACSD-63793_2.4.7-p3_v2.patch patch:

diff --git a/vendor/magento/module-import-export/Block/Adminhtml/Import/Edit/Form.php b/vendor/magento/module-import-export/Block/Adminhtml/Import/Edit/Form.php
index bf394c9ed0c42..11eaaa1a4a202 100644
--- a/vendor/magento/module-import-export/Block/Adminhtml/Import/Edit/Form.php
+++ b/vendor/magento/module-import-export/Block/Adminhtml/Import/Edit/Form.php
@@ -1,7 +1,7 @@
 <?php
 /**
- * Copyright © Magento, Inc. All rights reserved.
- * See COPYING.txt for license details.
+ * Copyright 2013 Adobe
+ * All Rights Reserved.
  */
 namespace Magento\ImportExport\Block\Adminhtml\Import\Edit;
 
@@ -264,6 +264,16 @@ protected function _prepareForm()
                 'value' => '',
             ]
         );
+        $fieldset->addField(
+            '_import_history_id',
+            'hidden',
+            [
+                'name' => '_import_history_id',
+                'label' => __('Import History id'),
+                'title' => __('Import History id'),
+                'value' => '',
+            ]
+        );
         $fieldsets['upload'] = $fieldset;
         $form->setUseContainer(true);
         $this->setForm($form);
diff --git a/vendor/magento/module-import-export/Controller/Adminhtml/Import/Validate.php b/vendor/magento/module-import-export/Controller/Adminhtml/Import/Validate.php
index c388851edcbe4..bf0758bb2105c 100644
--- a/vendor/magento/module-import-export/Controller/Adminhtml/Import/Validate.php
+++ b/vendor/magento/module-import-export/Controller/Adminhtml/Import/Validate.php
@@ -1,7 +1,7 @@
 <?php
 /**
- * Copyright © Magento, Inc. All rights reserved.
- * See COPYING.txt for license details.
+ * Copyright 2011 Adobe
+ * All Rights Reserved.
  */
 declare(strict_types=1);
 
@@ -50,6 +50,11 @@ public function execute()
                 $ids = $import->getValidatedIds();
                 if (count($ids) > 0) {
                     $resultBlock->addAction('value', Import::FIELD_IMPORT_IDS, $ids);
+                    $resultBlock->addAction(
+                        'value',
+                        '_import_history_id',
+                        $this->historyModel->getId()
+                    );
                 }
             } catch (\Magento\Framework\Exception\LocalizedException $e) {
                 $resultBlock->addError($e->getMessage());
diff --git a/vendor/magento/module-import-export/Model/History.php b/vendor/magento/module-import-export/Model/History.php
index 9a97367ba8453..ad0be7fbb8bc1 100644
--- a/vendor/magento/module-import-export/Model/History.php
+++ b/vendor/magento/module-import-export/Model/History.php
@@ -1,7 +1,7 @@
 <?php
 /**
- * Copyright © Magento, Inc. All rights reserved.
- * See COPYING.txt for license details.
+ * Copyright 2015 Adobe
+ * All Rights Reserved.
  */
 namespace Magento\ImportExport\Model;
 
@@ -9,33 +9,34 @@
  * Import history model
  *
  * @api
+ * @SuppressWarnings(PHPMD.CookieAndSessionMisuse)
  * @SuppressWarnings(PHPMD.CouplingBetweenObjects)
  * @SuppressWarnings(PHPMD.LongVariable)
  * @since 100.0.2
  */
 class History extends \Magento\Framework\Model\AbstractModel
 {
-    const HISTORY_ID = 'history_id';
+    public const HISTORY_ID = 'history_id';
 
-    const STARTED_AT = 'started_at';
+    public const STARTED_AT = 'started_at';
 
-    const USER_ID = 'user_id';
+    public const USER_ID = 'user_id';
 
-    const IMPORTED_FILE = 'imported_file';
+    public const IMPORTED_FILE = 'imported_file';
 
-    const ERROR_FILE = 'error_file';
+    public const ERROR_FILE = 'error_file';
 
-    const EXECUTION_TIME = 'execution_time';
+    public const EXECUTION_TIME = 'execution_time';
 
-    const SUMMARY = 'summary';
+    public const SUMMARY = 'summary';
 
-    const IMPORT_IN_PROCESS = 'In Progress';
+    public const IMPORT_IN_PROCESS = 'In Progress';
 
-    const IMPORT_VALIDATION = 'Validation';
+    public const IMPORT_VALIDATION = 'Validation';
 
-    const IMPORT_FAILED = 'Failed';
+    public const IMPORT_FAILED = 'Failed';
 
-    const IMPORT_SCHEDULED_USER = 0;
+    public const IMPORT_SCHEDULED_USER = 0;
 
     /**
      * @var \Magento\ImportExport\Helper\Report
@@ -122,7 +123,7 @@ public function addErrorReportFile($filename)
     public function updateReport(Import $import, $updateSummary = false)
     {
         if ($import->isReportEntityType()) {
-            $this->load($this->getLastItemId());
+            $this->load($import->getData('_import_history_id') ?? $this->getLastItemId());
             $executionResult = self::IMPORT_IN_PROCESS;
             if ($updateSummary) {
                 $executionResult = $this->reportHelper->getExecutionTime($this->getStartedAt());
diff --git a/vendor/magento/module-import-export/i18n/en_US.csv b/vendor/magento/module-import-export/i18n/en_US.csv
index cc1098841bab8..378daac3afa25 100644
--- a/vendor/magento/module-import-export/i18n/en_US.csv
+++ b/vendor/magento/module-import-export/i18n/en_US.csv
@@ -29,6 +29,7 @@ Import,Import
 "File to Import","File to Import"
 "Select File to Import","Select File to Import"
 "Images File Directory","Images File Directory"
+"Import History id","Import History id"
 "For Type ""Local Server"" use relative path to &lt;Magento root directory&gt;/var/import/images, e.g. <i>product_images</i>, <i>import_images/batch1</i>.<br><br>For example, in case <i>product_images</i>, files should be placed into <i>&lt;Magento root directory&gt;/var/import/images/product_images</i> folder.","For Type ""Local Server"" use relative path to &lt;Magento root directory&gt;/var/import/images, e.g. <i>product_images</i>, <i>import_images/batch1</i>.<br><br>For example, in case <i>product_images</i>, files should be placed into <i>&lt;Magento root directory&gt;/var/import/images/product_images</i> folder."
 "Download Sample File","Download Sample File"
 "Please correct the data sent value.","Please correct the data sent value."

As noted earlier, there’s more discussion and info on GitHub Issue #37593.

How to List/Change/Activate Different Service Versions on Fastly for Adobe Commerce

2025-11-23T00:00:00-00:00

There are times when you need to either list or activate an old version of your
Fastly configuration but you may not be able to access your Magento/Adobe Commerce
control panel to do so.

If you need to list versions of your service, you can do so using the Fastly CLI
tool using:

fastly service-version list -t API_KEY -s SERVICE_ID

This will return you a list of service versions and their associated dates:

NUMBER  ACTIVE  STAGED  LAST EDITED (UTC)
1        false    false    2020-06-09 17:25
2        false    false    2020-06-09 21:01
3        false    false    2020-06-10 00:08
4        false    false    2020-06-10 02:27
5        false    false    2020-06-10 06:40
6        false    false    2020-06-10 09:02
7        false    false    2020-06-11 16:07
8        false    false    2020-06-25 20:02
9        false    false    2020-06-25 20:02
10       false    false    2020-07-13 08:42
11       false    false    2020-07-13 08:43
12       false    false    2020-07-22 10:09
13       false    false    2020-07-28 20:07
14       false    false    2020-07-28 19:51
15       false    false    2020-08-03 12:04
16       false    false    2020-09-29 20:04
17       false    false    2020-10-02 12:03
18       false    false    2020-10-29 21:32
19       false    false    2020-12-01 12:03
20       false    false    2021-01-07 12:57
21       false    false    2021-01-07 12:58
22       false    false    2021-01-07 12:59
23       false    false    2021-01-07 13:01
24       false    false    2021-01-30 08:05
25       false    false    2021-02-16 10:41
26       false    false    2021-02-17 09:17
27       false    false    2021-03-18 14:29
28       false    false    2021-03-31 08:04
29       false    false    2021-05-06 10:14
30       false    false    2021-05-10 10:08
31       false    false    2021-05-10 10:09
32       false    false    2021-05-12 14:01
33       false    false    2021-05-30 08:05
34       false    false    2021-06-07 21:07
35       false    false    2021-06-07 21:19
36       false    false    2021-07-16 10:08
37       false    false    2021-07-16 11:11
38       false    false    2021-07-29 08:04
39       false    false    2021-09-27 08:05
40       false    false    2021-11-10 16:15
41       false    false    2021-11-11 17:39
42       false    false    2021-11-15 11:52
43       false    false    2021-11-26 08:04
44       false    false    2021-12-09 11:28
45       false    false    2022-01-25 08:04
46       false    false    2022-01-28 09:34
47       false    false    2022-02-08 16:30
48       false    false    2022-03-08 10:46
49       false    false    2022-03-26 08:04
50       false    false    2022-03-29 01:53
51       false    false    2022-04-19 12:27
52       false    false    2022-05-18 08:30
53       false    false    2022-05-25 08:04
54       false    false    2022-07-24 08:03
55       false    false    2022-07-24 08:03
56       false    false    2022-08-17 13:23
57       false    false    2022-09-20 08:21
58       false    false    2022-09-22 08:04
59       false    false    2022-09-27 13:47
60       false    false    2022-09-27 13:56
61       false    false    2022-09-28 09:11
62       false    false    2022-10-03 09:02
63       false    false    2022-10-19 13:56
64       false    false    2022-11-21 08:02
65       false    false    2022-12-06 05:43
66       false    false    2022-12-08 15:14
67       false    false    2022-12-13 09:31
68       false    false    2022-12-13 09:34
69       false    false    2022-12-22 16:00
70       false    false    2022-12-22 16:04
71       false    false    2023-01-20 08:03
72       false    false    2023-01-27 12:45
73       false    false    2023-02-23 16:15
74       false    false    2023-03-21 08:02
75       false    false    2023-04-21 16:07
76       false    false    2023-04-21 16:34
77       false    false    2023-04-21 16:46
78       false    false    2023-04-21 18:25
79       false    false    2023-04-21 20:16
80       false    false    2023-04-22 11:53
81       false    false    2023-04-22 12:03
82       false    false    2023-04-24 09:39
83       false    false    2023-04-24 09:55
84       false    false    2023-04-24 09:56
85       false    false    2023-04-24 10:00
86       false    false    2023-04-24 10:07
87       false    false    2023-04-27 13:05
88       false    false    2023-04-27 13:07
89       false    false    2023-05-02 08:35
90       false    false    2023-05-02 10:22
91       false    false    2023-05-20 08:04
92       false    false    2023-05-24 17:20
93       false    false    2023-05-25 09:28
94       false    false    2023-05-31 18:13
95       false    false    2023-07-06 12:34
96       false    false    2023-07-19 08:05
97       false    false    2023-08-15 09:41
98       false    false    2023-08-15 09:41
99       false    false    2023-08-15 10:51
100      false    false    2023-08-15 10:52
101      false    false    2023-08-15 14:31
102      false    false    2023-08-22 07:08
103      false    false    2023-08-22 13:45
104      false    false    2023-08-22 15:21
105      false    false    2023-08-22 15:26
106      false    false    2023-08-22 15:25
107      false    false    2023-08-22 15:25
108      false    false    2023-09-06 21:37
109      false    false    2023-08-22 15:26
110      false    false    2023-09-15 13:28
111      false    false    2023-09-15 13:38
112      false    false    2023-09-17 08:04
113      false    false    2023-09-23 05:36
114      false    false    2023-09-23 05:49
115      false    false    2023-09-23 06:12
116      false    false    2023-09-25 06:09
117      false    false    2023-10-05 05:58
118      false    false    2023-10-05 06:35
119      false    false    2023-10-05 09:06
120      false    false    2023-10-05 09:08
121      false    false    2023-10-05 09:09
122      false    false    2023-10-05 09:29
123      false    false    2023-10-27 07:20
124      false    false    2023-11-07 14:24
125      false    false    2023-11-16 08:04
126      false    false    2023-11-20 12:45
127      false    false    2023-12-12 08:05
128      false    false    2023-12-13 17:57
129      false    false    2024-01-15 08:04
130      false    false    2024-03-01 10:44
131      false    false    2024-03-15 08:05
132      false    false    2024-04-10 14:16
133      false    false    2024-04-10 14:20
134      false    false    2024-05-02 18:39
135      false    false    2024-05-14 08:05
136      false    false    2024-05-17 10:23
137      false    false    2024-05-17 10:26
138      false    false    2024-05-17 10:31
139      false    false    2024-05-17 10:34
140      false    false    2024-05-17 12:24
141      false    false    2024-05-17 12:25
142      false    false    2024-05-17 12:28
143      false    false    2024-05-21 09:11
144      false    false    2024-05-21 09:17
145      false    false    2024-05-21 09:30
146      false    false    2024-05-21 09:30
147      false    false    2024-05-21 12:22
148      false    false    2024-06-03 10:03
149      false    false    2024-06-19 06:07
150      false    false    2024-06-28 06:55
151      false    false    2024-07-05 12:05
152      false    false    2024-07-08 07:14
153      false    false    2024-07-08 11:52
154      false    false    2024-07-09 07:25
155      false    false    2024-07-13 08:03
156      false    false    2024-07-19 06:28
157      false    false    2024-07-19 06:57
158      false    false    2024-07-22 11:32
159      false    false    2024-08-16 12:39
160      false    false    2024-09-11 08:04
161      false    false    2024-09-25 17:58
162      false    false    2024-10-14 21:41
163      false    false    2024-10-22 07:29
164      false    false    2024-10-22 22:09
165      false    false    2024-11-10 08:04
166      false    false    2024-11-13 14:30
167      false    false    2024-11-22 08:03
168      false    false    2024-12-17 08:42
169      false    false    2024-12-22 03:13
170      false    false    2024-12-22 03:58
171      false    false    2025-01-09 08:04
172      false    false    2024-12-22 03:53
173      false    false    2025-01-29 21:54
174      false    false    2025-01-29 21:54
175      false    false    2025-02-13 14:20
176      false    false    2025-03-10 08:05
177      false    false    2025-03-21 08:44
178      false    false    2025-03-21 12:03
179      false    false    2025-03-21 15:12
180      false    false    2025-03-21 15:13
181      false    false    2025-03-25 23:02
182      false    false    2025-05-08 16:36
183      false    false    2025-05-09 08:03
184      false    false    2025-05-15 16:18
185      false    false    2025-05-21 11:16
186      false    false    2025-05-24 13:15
187      false    false    2025-06-11 13:24
188      false    false    2025-07-08 08:03
189      false    false    2025-07-29 16:18
190      false    false    2025-08-06 16:26
191      false    false    2025-09-05 14:48
192      false    false    2025-09-06 08:04
193      false    false    2025-09-08 17:38
194      false    false    2025-09-11 22:18
195      false    false    2025-09-22 07:00
196      false    false    2025-09-22 23:59
197      false    false    2025-10-14 16:04
198      false    false    2025-10-20 12:13
199      false    false    2025-10-20 13:25
200      false    false    2025-10-21 12:16
201      false    false    2025-11-05 08:04
202      false    false    2025-11-20 10:58
203      false    false    2025-11-20 10:46
204      false    false    2025-11-20 11:30
205      false    false    2025-11-20 11:53
206      true     false    2025-11-20 11:53

From here, you can activate one of those versions by running this command:

fastly service-version activate --version=204 -t API_KEY -s SERVICE_ID

You can fetch both your API_KEY key and SERVICE_ID from the file mounted at:

/mnt/shared/fastly_tokens.txt

in your Commerce Cloud cluster if you don’t have the tokens to hand.

Flush Fastly CDN via CLI/Command Line for Adobe Commerce Projects

2025-11-16T00:00:00-00:00

If you ever need the ability to flush Fastly but can’t acess an Adobe Commerce
admin panel to do so, you can run the following curl request:

curl -H "Fastly-Key: API_KEY” -X POST "https://api.fastly.com/service/SERVICE_ID/purge_all"

You can fetch both your API_KEY key and SERVICE_ID from the file mounted at:

/mnt/shared/fastly_tokens.txt

in your Commerce Cloud cluster.

Adobe Commerce Deployment issue 'does not support Valkey backend model'

2025-04-13T00:00:00-00:00

If you’ve recently deployed one of the latest release lines of Magento on Commerce Cloud and have:

REDIS_BACKEND: '\Magento\Framework\Cache\Backend\RemoteSynchronizedCache'

[2025-04-08T10:29:29.486342+00:00] 
ERROR: Magento version '2.4.7.0-patch5' does not support Valkey backend 
model '\Magento\Framework\Cache\Backend\RemoteSynchronizedCache'

There is a Github issue for this problem. That
latest release of ece-tools does have a fix for the issue albeit not a great fix.

Adobe Commerce B2B upgrade issue 'REGEXP_LIKE does not exist'

2025-04-13T00:00:00-00:00

If you’re using Adobe’s B2B extension and recently tried upgrading then you likely hit this issue:

W: Unable to apply data patch Magento\Company\Setup\Patch\Data\SetComp
W: anyForStructure for module Magento_Company. Original exception message: SQL
W: STATE[42000]: Syntax error or access violation: 1305 FUNCTION abcdefghijk 
W: .REGEXP_LIKE does not exist, query was: UPDATE `company_structure` SET `com
W: pany_id` = ? WHERE (REGEXP_LIKE(path, '^7(/.+)?$'))

I sent this in to Adobe’s Cloud support and got this Absolute gem of a response from support:

We investigated this further and please find more details regarding this error below:

W: Unable to apply data patch Magento\Company\Setup\Patch\Data\SetComp
W: anyForStructure for module Magento_Company. Original exception message: SQL
W: STATE[42000]: Syntax error or access violation: 1305 FUNCTION abcdefghijk
W: .REGEXP_LIKE does not exist, query was: UPDATE company_structure SET com W: pany_id = ? WHERE (REGEXP_LIKE(path, ‘^7(/.+)?$’))

The Adobe Commerce version 2.4.6-p10 uses the function “REGEXP_LIKE” in the file \Magento\Company\Setup\Patch\Data\SetCompanyForStructure.

The function “REGEXP_LIKE” is not supported in hashtag MariaDB which is the default Database used by Adobe Commerce.

No idea how this one got through QA.

If you pin the version of the B2B extension to 1.4.2-p3 it’ll work or you could patch out that function.

Laravel 12 $request->getContentType() does not exist

2025-04-12T00:00:00-00:00

If you use Laravel’s Request object and had previously used the $request->getContentType() method
then in Laravel 12, it’s been changed to $request->getContentTypeFormat():

Old:


/**
 *
 */
class ApiRequestHandler
{
    /**
     * @param Illuminate\Http\Request $request
     */
    public function __invoke(\Illuminate\Http\Request $request)
    {
        if ($request->getContentType() === 'form') {
          // Do something
        }

    }
}

New:


/**
 *
 */
class ApiRequestHandler
{
    /**
     * @param Illuminate\Http\Request $request
     */
    public function __invoke(\Illuminate\Http\Request $request)
    {
        if ($request->getContentTypeFormat() === 'form') {
          // Do something
        }

    }
}

Set Website/Store Assignment via Magento 2 API

2025-02-28T00:00:00-00:00

If you need to update the website a product is assigned to in Magento via the
REST API, you can do this fairly easily.

You can get the current websites assigned by running this query:

curl "https://www.mystore.co.uk/rest/all/V1/products/MY-PRODUCT-SKU" \
     -H 'Content-Type: application/json' \
     -H 'Authorization: Bearer [token]'

This will return a JSON payload. Within the payload you’ll find:

  "id": 5687,
  "sku": "MY-PRODUCT-SKU",
  "name": "My Product",
  "attribute_set_id": 4,
  "price": 9.99,
  "status": 1,
  "visibility": 4,
  "type_id": "simple",
  "created_at": "2024-03-07 13:45:29",
  "updated_at": "2025-02-24 14:20:09",
  "weight": 1,
  "extension_attributes": {
    "website_ids": [
      1,
      3
    ],
  }

The key part you’re looking for there is website_ids within
extension_attributes which will give you the currently assigned website IDs.

To set this data, you can make a PUT request:

curl -X "PUT" "https://www.mystore.co.uk/rest/all/V1/products/MY-PRODUCT-SKU" \
     -H 'Content-Type: application/json' \
     -H 'Authorization: Bearer [token]' \
     -d $'{
  "product": {
    "extension_attributes": {
      "website_ids": [
        1,
        3,
        14,
        22
      ]
    }
  }
}'

Which will then set the product to be in the given websites.

Can't place the order error for Paypal Express (when Stripe is also installed) with Magento 2

2025-01-10T00:00:00-00:00

This week I needed to debug an issue when Paypal Express was being used and it was throwing a “We can’t place the order”.

If you’ve ever looked into the Paypal Express controller within Magento then you’ll see that it’s effectively a catch all when something in the process fails.

If you take a look at vendor/magento/module-paypal/Controller/Express/AbstractExpress/PlaceOrder.php:

141        } catch (ApiProcessableException $e) {
142            $this->_processPaypalApiError($e);
143        } catch (LocalizedException $e) {
144            $this->processException($e, $e->getRawMessage());
145        } catch (\Exception $e) {
146            $this->processException($e, 'We can\'t place the order.');
147        }

This bug presents itself when the user follows this process:

Order an item that doesn’t have free shipping
Guest user
Using a PHP8.2 compatible version of Stripe stripe/stripe-payments extension, in this case it was v3.5.16

If you imagine the customer going through this flow:

Add item to cart that doesn’t have free shipping, let’s say £10.00 inc VAT.
User uses a shortcut button to go and pay via Paypal Express
The user authorises £10.00 at Paypal and then returns to the order review page
At this point, they need to select shipping which we’ll say is £5 inc VAT.
When the user now tries to place the order, the total required is £15.00 - not the pre authorised £10.

This causes Paypal to reject the transaction:

Exception #0 (Exception): PayPal gateway has rejected request. This transaction couldn't be completed. Please redirect your customer to PayPal (#10486: This transaction couldn't be completed).

In normal working order, the Paypal extension then throws a ApiProcessableException exception, which is then caught by line 141 in the code example above.

Now, this is where it gets problematic. Stripe have a plugin located at vendor/stripe/module-payments/Plugin/Sales/Model/Service/OrderService.php which has an around on the place() method:

public function aroundPlace($subject, \Closure $proceed, $order)
{
    try
    {
        if (!empty($order) && !empty($order->getQuoteId()))
        {
            $this->quoteHelper->quoteId = $order->getQuoteId();
        }

        $savedOrder = $proceed($order);

        return $this->postProcess($savedOrder);
    }
    catch (\Exception $e)
    {
        $helper = $this->helperFactory->create();
        $msg = $e->getMessage();

        if ($helper->isAuthenticationRequiredMessage($msg))
            throw $e;
        else
            $helper->dieWithError($e->getMessage(), $e);
    }
}

The problem here is that Stripe are assuming they’re the only paypal in play that may throw an exception. They’re just blanket catching all exceptions and hoovering them up which then catches Paypal’s ApiProcessableException exception as well - meaning the user is never redirected to Paypal to authorise the new amount of £15.00.

For a company as big as Stripe, I’m genuinely surprised that they haven’t written this better or even checked that the current payment method in the process is actually a Stripe one. Defensive programming it is not.

The patch is fairly simple:

diff --git a/vendor/stripe/module-payments/Plugin/Sales/Model/Service/OrderService.php b/vendor/stripe/module-payments/Plugin/Sales/Model/Service/OrderService.php
index 1fe4d32e2..bdd6cb0f7 100644
--- a/vendor/stripe/module-payments/Plugin/Sales/Model/Service/OrderService.php
+++ b/vendor/stripe/module-payments/Plugin/Sales/Model/Service/OrderService.php
@@ -38,6 +38,10 @@ class OrderService
 
     public function aroundPlace($subject, \Closure $proceed, $order)
     {
+        if (!str_contains($order->getPayment()->getMethod(), "stripe_")) {
+            return $proceed($order);
+        }
+
         try
         {
             if (!empty($order) && !empty($order->getQuoteId()))

Just a check to make sure that the payment method being used is actually a Stripe one. This prevents the hoovering up on non-Stripe payment methods.

There’s a few people who have bumped into similar issues on Github too https://github.com/magento/magento2/issues/34412.

Hyva Not Generating app/etc/hyva-themes.json on Deployment for Magento 2.4.7

2024-10-27T00:00:00-00:00

Update: Hyva are now recommending Adobe Commerce Cloud customers commit their app/etc/hyva-themes.json file into the repo.

The app/etc/hyva-themes.json file
The file needs to be generated after Magento is installed, so it can’t be run during the build phase. This means, for Adobe Commerce Cloud projects, the app/etc/
hyva-themes.json file needs to be commited to the repository.
Otherwise the file will be missing during the compilation of the styles.

and they’ve update the docs accordingly.

If you don’t wish the commit that file into your repo, the initial solution in this post still works.

At work this week I was updating the deployment scripts for one of our projects
when I realised that Hyva wasn’t generating the app/etc/hyva-themes.json
file on deployment which then caused the pipeline to fail.

This was due to two different things, there’s a bug in Hyva for Magento 2.4.7
where it isn’t properly triggering a generation. That problem is fixed in an
existing Merge Request on Hyva’s Gitlab which solves the problem by triggering generation when ConfigImportProcessor is intercepted (see MR for more).

The second issue is more of a conceptual issue. The default instruction for
deploying on Adobe Commerce Cloud won’t work out the box for most Commerce
Cloud deployments due to the ordering of deployment comamnds (where
app/etc/hyva-themes.json isn’t tracked in the repo).

This is because the Hyva generation via NPM runs before one of the commands to
trigger the writing of app/etc/hyva-themes.json. At the time
npm run build-prod fires - there isn’t a app/etc/hyva-themes.json written
so the deployment will fail with something like this:

> npm run build && npm run output-success


> @hyva-themes/magento2-default-theme@3.0.0 build
> NODE_ENV=production npx tailwindcss --postcss -i tailwind-source.css -o ../css/styles.css --minify

W: node:fs:603
W:   handleErrorFromBinding(ctx);
W:   ^
W: 
W: Error: ENOENT: no such file or directory, open '/app/app/etc/hyva-themes.json'
W:     at Object.openSync (node:fs:603:3)
W:     at Object.readFileSync (node:fs:471:35)
W:     at Object.<anonymous> (/app/app/design/frontend/Deploy/project/web/tailwind/node_modules/@hyva-themes/hyva-modules/src/index.js:107:19)
W:     at Module._compile (node:internal/modules/cjs/loader:1256:14)
W:     at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
W:     at Module.load (node:internal/modules/cjs/loader:1119:32)
W:     at Module._load (node:internal/modules/cjs/loader:960:12)
W:     at Module.require (node:internal/modules/cjs/loader:1143:19)
W:     at require (node:internal/modules/cjs/helpers:121:18)
W:     at Object.<anonymous> (/app/app/design/frontend/Deploy/project/web/tailwind/postcss.config.js:1:38) {
W:   errno: -2,
W:   syscall: 'open',
W:   code: 'ENOENT',
W:   path: '/app/app/etc/hyva-themes.json'
W: }
W: 
W: Node.js v18.17.1

E: Error building project: Step failed with status code 1.

E: Error: Unable to build application, aborting.

This makes sense, give the bug in Hyva with 2.4.7 not trigger the writing of
the file, it can’t read from it as its not there.

You can work around this by setting your build hook in .magento.app.yaml to
disable/enable one of the Hyva core extensions which triggers the writing of
the required app/etc/hyva-themes.json:

hooks:
  # We run build hooks before your application has been packaged.
  build: |
    set -e
    composer --no-ansi --no-interaction install --no-progress --prefer-dist --optimize-autoloader --no-dev
    
    echo 'Generating app/etc/hyva-themes.json'
    bin/magento module:disable Hyva_Theme && bin/magento module:enable Hyva_Theme
    
    unset NPM_CONFIG_PREFIX
    curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
    export NVM_DIR="$HOME/.nvm"    
    [ -s "$NVM_DIR/nvm.sh"  ] &&. "$NVM_DIR/nvm.sh"
    nvm install 18.17

    echo 'Generating Hyva styles...'
    mkdir -p app/design/frontend/Deploy/project/web/css/
    npm --prefix app/design/frontend/Deploy/project/web/tailwind/ install
    npm --prefix app/design/frontend/Deploy/project/web/tailwind/ run build-prod
    rm -rf app/design/frontend/Deploy/project/web/tailwind/node_modules/
    rm -rf ~/.nvm/    

    php ./vendor/bin/ece-tools run scenario/build/generate.xml
    php ./vendor/bin/ece-tools run scenario/build/transfer.xml
  deploy: |
    php ./vendor/bin/ece-tools run scenario/deploy.xml
  post_deploy: |
    php ./vendor/bin/ece-tools run scenario/post-deploy.xml

This way, it’ll write the app/etc/hyva-themes.json file and then generate
your Tailwind styles before the main setup:static-content:deploy runs.

If you’re running setup:upgrade at some point in your pipepine before you
generate your tailwind styles then you likely won’t run into this.

Klarna Checkout Won't Install on Magento 2.4.4

2022-05-24T00:00:00-00:00

If you’re using the Klarna Checkout package (klarna/m2-checkout) then when you
upgrade to Magento 2.4.4 it won’t install due to a conflict with Monolog
versions. As per a merchant email a short while ago, you need to change over to
the new klarna/m2-checkout package.

You can switch over using:

composer remove klarna/m2-checkout
composer require klarna/m2-klarna

How to Re-run a Composer Script After install/create-project

2022-05-23T00:00:00-00:00

You can re-run a composer script by using this syntax:

composer run-script post-create-project-cmd

Magento 2.4.4 Admin Login Redirect Loop/Not Working

2022-04-25T00:00:00-00:00

If you hit a redirect loop in the admin panel after upgrading to 2.4.4 then you
like had the configuration option system/security/max_session_size_admin set
to 0 to fix an issue with 2.4.3 and session size. Setting it to 0 in 2.4.4
doesn’t work and causes the session to not get written.

If you set it to an actual upper limit then it’ll work:

bin/magento config:set system/security/max_session_size_admin 5120000
bin/magento config:set system/security/max_session_size_storefront 5120000

Clear the cache and you should be able to login afterwards.

Create Repeating Google Calendar Event for Last Working Day of Month

2022-04-21T00:00:00-00:00

I think Google Calendar is brilliant however
one type of repeating event it does not support is the last working day of the
month. There is the option to import a custom ICS file though and within the
specification of that file format you can get pretty much close to last working
day (there may be some exceptions where this doesn’t work, such a bank holidays
but it is close enough).

Create a new file, ie last-working-day.ics and put this inside it:

BEGIN:VCALENDAR
VERSION:2.0
BEGIN:VEVENT
RRULE:FREQ=MONTHLY;INTERVAL=1;WKST=MO;BYDAY=MO,TU,WE,TH,FR;BYSETPOS=-1
SUMMARY:Last working day
DTSTART;VALUE=DATE:20220421
SEQUENCE:0
DESCRIPTION:Repeating last working day event
END:VEVENT
END:VCALENDAR

Save that and then import into Google Calendar
and it’ll create you a new event that repeats on the last working day of the
month. If you need help importing into Google calendar then see this page in Google’s help docs.

Create Repeating Google Calendar Event for Last Specific Day of Month

2022-04-20T00:00:00-00:00

I think Google Calendar is brilliant however
one type of repeating event it does not support is the last specific day of the
month. There is the option to import a custom ICS file though and within the
specification of that file format you can do it.

Create a new file, ie last-wed-of-month.ics and put this inside it:

BEGIN:VCALENDAR
VERSION:2.0
BEGIN:VEVENT
RRULE:FREQ=MONTHLY;INTERVAL=1;WKST=MO;BYDAY=WE;BYSETPOS=-1
SUMMARY:Last Wednesday of Month
DTSTART;VALUE=DATE:20220421
SEQUENCE:0
DESCRIPTION:Last Wednesday of the month
END:VEVENT
END:VCALENDAR

If you need to do a different day to Wednesday, change BYDAY=WE to match the day you require (one of SU,MO,TU,WE,TH,FR,SA)

Create Repeating Google Calendar Event for Last Day of Month

2022-04-19T00:00:00-00:00

I think Google Calendar is brilliant however
one type of repeating event it does not support is the last day of the
month. There is the option to import a custom ICS file though and within the
specification of that file format you can do it.

Create a new file, ie last-day-of-month.ics and put this inside it:

BEGIN:VCALENDAR
VERSION:2.0
BEGIN:VEVENT
RRULE:FREQ=MONTHLY;INTERVAL=1;BYSETPOS=-1;BYDAY=SU,MO,TU,WE,TH,FR,SA
SUMMARY:Last day of the month
DTSTART;VALUE=DATE:20220421
SEQUENCE:0
DESCRIPTION:Last day of the month
END:VEVENT
END:VCALENDAR

Using logrotate as a non-root user

2022-01-08T00:00:00-00:00

There may be situations when you need to use logrotate without having root
access to the server that you’re dealing with. In these situations you can
point logrotate at a state file in your home directory.

First, create your logrotate configuration file with something like:

/var/log/messages {
    rotate 5
    weekly
}

and then save to ~/logrotate.conf. Next, touch your state file:

touch ~/.logrotate-state

and then you can run logrotate either via SSH or via cron:

/sbin/logrotate —state ~/.logrotate-state ~/.logrotate.conf

This is useful for providers that don’t give access to logrotate directly
but the binary is present to use.

How to Install PHP 7.2 (and 7.1, 7.4 etc) on MacOS Monterey

2022-01-07T00:00:00-00:00

As PHP older versions have now been deprecated upstream you will encounter
errors when trying to to install via Homebrew. For example, attempting to
install PHP 7.2 will give the following:

Error: php@7.2 has been disabled because it is deprecated upstream!

There’s a tap you can use at shivammathur/php which has all of the older
versions that are still installable.

First, tap the repo:

brew tap shivammathur/php

and then install the PHP version you want:

brew install shivammathur/php/php@7.2

Give it a few minutes to go off and do it’s thing and you’ll then have the
desired version running locally. I can understand why the PHP deprecated the
older versions but there’s an awful lot of developers out there who will
still need to run and install older versions of PHP due to constraints they
are under.

Easily Create Windows 10/8/7 (or any other OS) Bootable USB Drives

2021-08-09T00:00:00-00:00

If you ever need to create a bootable USB installer, the most reliable way that
I’ve found to do so on a Windows machine is using Rufus.

It works every single time and I’ve yet to encounter any issues with a drive
created using the tool.

I last used it to create a Windows 10 bootable drive after their own instructions
didn’t work whereas Rufus worked flawlessly.

Wordpress Docker - How to set memory_limit, max_upload_size and other resource limits

2021-07-14T00:00:00-00:00

To set max_upload_size and post_max_size to set in the official docker image, open a bash session and run:

root@f125876a0568:/var/www/html# echo “php_value upload_max_filesize 256M” >> .htaccess
root@f125876a0568:/var/www/html# cat .htaccess
# snipped
php_value post_max_size 256M
php_value upload_max_filesize 256M

Using >> in the echo command will append the line to the existing content present.

How to Stop Yourself Bricking Your Mac - Disable Secure Boot

2021-07-14T00:00:00-00:00

If you’re lucky to have a newer Mac with a T2 chip in it, at some point you’re likely going to reinstall the OS to freshen things up.

If “Allow booting from external media” is disabled (see image below) in Startup Security and you use Internet Recovery which then fails at any point (past the point of changing disk partitions) - when hen you then reboot your Mac, it’ll be bricked.

Internet Recovery can fail for many reasons - poor internet connection, checksum failure or any other number of things you don’t control.

The recovery partition will be external media and it’ll refuse to write to the disk in the machine. As expected, other external drives won’t boot either.

Like me, your Macbook will have to go to for a trip to Apple. The lovely folks at the Genius bar were stumped and the likely remedy at the moment appears to be a logic board replacement.

To prevent this happening to you, I would highly recommend that you do the following unless you have very good reason not to:

To allow your Mac to use an external startup disk:

Open Startup Security Utility.
Select “Allow booting from external media”.

There instructions are present on Apples site linked above.

Security is useful, everyone would agree but the default for this surely has to be off/allowing external media. At no point in having this Mac did I set that (I didn’t even know the option existed).

Magento 2 - How to delete all special prices

2021-05-18T00:00:00-00:00

If you ever need to remove all special/sale prices from a Magento 2 installation, you can use the following SQL to do so.

First step, back up your databases if you haven’t already. Specifically the catalog_product_entity_decimal table.

Then run the following SQL:

DELETE FROM catalog_product_entity_decimal
WHERE attribute_id IN (SELECT attribute_id FROM eav_attribute WHERE attribute_code = ‘special_price’);

If you want to see what it’s going to delete beforehand, run this query instead:

SELECT * FROM catalog_product_entity_decimal
WHERE attribute_id IN (SELECT attribute_id FROM eav_attribute WHERE attribute_code = ‘special_price’);

After it has completed, run bin/magento indexer:reindex to reindex the changed prices.

macOS Big Sur - Installing pip for Python

2020-11-15T00:00:00-00:00

The new version of macOS doesn’t appear to have pip installed by default, you can install it by running this command in a Terminal window:

sudo easy_install install pip

Note: Make sure that you have version 12.2 of Xcode installed and that in Xcode -> Preferences -> Locations -> Command Line Tools is set to Xcode 12.2 too.

A quick way to truncate files on Mac and Linux systems

2020-10-29T00:00:00-00:00

A short-hand method of truncating files is by using this syntax:

: > /path/to/filename.txt

Once it has ran then the files content will be removed and the file itself left in place.

Show user groups on Linux

2020-08-29T00:00:00-00:00

On most standard Linux operating systems you can view the current users’
group membership with:

id -nG

or if you’re currently logged in as root then you can just tag the user
that you want to view on the end of the command. For example, if I’m logged in
as root and want to see the groups that john is a member of:

id -nG john

which will then output:

root@mimas:~# id -nG john
john docker

A neat app for team safety checkins and announcements

2020-07-31T00:00:00-00:00

Clever little iOS app for businesses/teams called Mustr.

It can broadcast critical/urgent messages to your team and take roll calls to make sure everyone’s healthy, safe and secure.

🔗 https://t.co/HdOah1fKHa #ios #app #teams #remotework #remote #safety

— Scott (@ssxio) July 30, 2020

Keeping Your Privacy: How to Set-up Your Own VPN Using Cloud Providers

2020-06-14T00:00:00-00:00

This post is part of a series called “Keeping Your Privacy”.

There are a hundred and one reasons to run your own VPN - the biggest and most commonly referenced is privacy. You can control your own privacy when using public networks or public wifi for example. If you’re connecting via your own VPN service then you’re a lot more hidden.

In the current climate of protests you would be able to protect your communications over 4G mobile data networks if you had your own VPN.

Our VPN Choice

We’re going to use Streisand for our VPN. As we can read from their Github repository:

Streisand sets up a new server running your choice of WireGuard, OpenConnect, OpenSSH, OpenVPN, Shadowsocks, sslh, Stunnel, or a Tor bridge. It also generates custom instructions for all of these services. At the end of the run you are given an HTML file with instructions that can be shared with friends, family members, and fellow activists.

So, out of the box we get OpenVPN setup and configured, along with socks proxies and some other neat features with minimal configuration. The actual reason I selected Streisand is because I wanted something OpenVPN compatible and once the server has finished setting itself up - it writes a folder full of documentation for each service that explains how to configure your devices and start using the services offered

Select a Cloud Provider

We’re going to need a cloud provider to run your VPN within. My advice is the same for when you’re running any kind of server on other peoples hardware - pick a company you trust and feel comfortable with. Over the years I’ve launched projects both small and large. Some clients picked AWS and some picked DigitalOcean. If you haven’t got an account set up, you can use one of these referral links to get free credit (and I’ll get a small amount of credit too):

DigitalOcean
Linode
Vultr
AWS LightSail

Next, the process for creating a new server varies between platform. We’re going to create a new Ubuntu 16.10 server (due to its great compatibility with Streisand).

Linode, AWS Lightsail and Vultr videos will be added shortly.

After the server creation is completed and you’ve got a running Ubuntu 16.10 server, copy the IP address and we can move to the installation stage.

Pick Your Subdomain and Configure DNS

You’re going to need a fully set up DNS record for your new VPN server, if you own bob.com, then vpn.bob.com wouldn’t be a bad choice. You’ll need this during set-up of Streisand so make sure your DNS A record exists and points at your new servers IP address otherwise installation will fail and SSL certificates won’t be issued properly.

Streisand Installation

Follow these steps:

SSH in as root to your server ssh root@your-server-ip (or via the DNS name for your box as you should have set that up already)
Run apt-get install git python3 python3-venv
Clone the Streisand repo: git clone https://github.com/StreisandEffect/streisand.git && cd streisand
Run ./util/venv-dependencies.sh ./venv and it’ll report anything more you need to do. Follow any instructions it gives you
Run source ./venv/bin/activate
run ./streisand and follow prompts. Defaults are fine to all answers.

Device Configuration

If you SFTP to your new server, checkout the directory that you cloned Streisand into and there will be a directory called generated-docs with configuration information and instructions. Supports Mac OS, iOS, Android, Windows and more.

Follow on:

It’d be wise to set the reverse-dns entry for your servers IP address and also have a look around for other VPN setup packages - I’ve heard good things about Algo along with Streisand.

Keeping Your Privacy: Face Blurring & EXIF Data

2020-06-11T00:00:00-00:00

This post is part of a series called “Keeping Your Privacy”.

It has been truly remarkable to see the protests around the world in the last two weeks. A message seems to be spreading and getting through to politicians and those in power at least seem to acknowledge that something has to change going forward.

The liability in the White House hasn’t exactly helped matters state side but with a white supremacist in the hot seat it’s not exactly a surprise. You don’t even have to think too far back to some of the early remarks on Charlottesville for an indication of who Trump really is. Anyhow, I digress as that isn’t the topic at hand.

There are a few things that people protesting need to remember from my experience and I’ve seen countless others say similar.

Keep your face covered, once these protests are over I have zero doubt that the police will try and start to identify people taking part.
The same goes for those with microphones/amplifiers. You’ll be even more of a target for being more active in the protest
By all means, take and share photos, videos and whatever else you want to but crucially cover faces. Traditionally people would blur these but I’m not so sure that is a good idea any more which I’ll elaborate more on below.

Blurred Lines

Over the years I’ve seen plenty of people blur photos and videos of people taking part in protests to hide their identity. After recently researching more into it and finding the work of Vladimir Yuzhikov - I would highly recommend covering with a big block of colour to hide peoples identities and no longing use blur as an effective privacy tool.

What I also feel strongly about is that while Vladimir was able to achieve these fantastic results with modest software and knowledge - I’m sure that law enforcement staff have a tool which will de-blur a lot quicker and provide immediate results.

Be Wary of

One last thought here is that I’ve seen a number of people on Twitter offering to help ‘cover up’ protesters faces and protect their privacy for you. While I’m sure most of these offers are genuine I couldn’t help the uneasy feeling I had - I’d bet that some of those offering help were likely police phishing around for photos and videos from the protests as they’re now collecting evidence which will likely lead to arrests in future.

In short, I’d stick with the same rule I have for sending money to people online - don’t do it unless you know them in person.

One Last Thing - EXIF

When you take a photo, your camera embeds certain data inside the photo file. This contains things like light levels, camera configuration and sometimes the physical location the photograph was taken in (some video recording hardware does this in a slightly different way - Go Pro for example).

Before you share your photos/videos online - I’d highly, highly recommend you strip the EXIF data from your files - at a minimum remove the location data.

For an iPhone, follow this iMore guide. Also this is easy on an iOS or Mac device, for other platforms I can imagine this being difficult to do.

Android has Metadata Cleaner but I’ve not personally tested that. For those on Windows, give ExifPurge a try. Linux users can use the brilliant exiftool to handle their images.

If you’re on a different platform then I’d suggest you try Googling for 'Remove Exif data from on ’ and see what you find.

Introducing a Series: Keeping Your Privacy

2020-06-10T00:00:00-00:00

Everyone is aware that privacy online is an important topic. It becomes even more important when you apply the context of the current protests and people needing to protect their identity.

I’m lucky that I’ve worked as a professional software developer for a long time and know how to configure, run and operate lots of different services. When the protests broke out worldwide I felt like I should use my knowledge to share with others how to protect themselves.

There’s a famous Desmond Tutu quote:

“If you are neutral in situations of injustice, you have chosen the side of the oppressor. If an elephant has its foot on the tail of a mouse, and you say that you are neutral, the mouse will not appreciate your neutrality.”

With doing a small thing to share knowledge with others, if that only helps one other person then it’s been worthwhile.

This page will act as the overview for this series and it starts with an explanation of blurring faces from your media and removing EXIF data.

Keeping Your Privacy: Face Blurring & EXIF Data
Keeping Your Privacy: How to Set-up Your Own VPN Using Cloud Providers [Coming Soon]

How to Install and Configure Docker on Ubuntu 20.04

2020-06-09T00:00:00-00:00

Create a non-root user to run your docker containers under:

adduser steve

Next, update the apt sources and install Docker:

apt install docker.io
systemctl enable —now docker

Now, add our new user to the docker user group so they can interact with the service:

usermod -aG docker steve

and finally, we’ll switch over to that user using su and confirm we can run a simple container:

su steve
docker run hello-world

That’ll print out the Docker welcome message, confirming that each step of the process succeeded. If you want to try something a little more interesting, you can boot an Ubuntu container with:

docker run -it ubuntu bash

-it tells Docker that you want to run this interactively and to assign a tty to you

Other than that, you can run pretty normal things within your Ubuntu container but keep in mine any changes you make or configuration you do won’t be persisted.

How to Configure and Use Hardware Yubikey with Mac OS SSH

2020-05-22T00:00:00-00:00

Due to recent updates to OpenSSH there is now a much simpler and easier way to use hardware tokens for SSH access.

Start by inserting your Yubikey into your Mac.

Create a new key ecdsa-sk which will reside on your machine but require your hardware token:

ssh-keygen -t ecdsa-sk -f ~/.ssh/id_ecdsa_sk -C “user@site.co”

It’ll ask you to enter a password to protect your key, then if touch compatible you’ll need to press the light when asked.

Copy the contents of ~/.ssh/id_ecdsa_sk.pub to the .ssh/authorized_keys file on your remote server and then you’re ready to test the connection works, login using your new identity:

ssh -i ~/.ssh/id_ecdsa_sk user@server.com

You’ll then be protected for your key password, then prompted to authorise the connection request by touching your hardware key:

It makes it easier to store the identity file in your .ssh/config file so you don’t have to type it out each time, use a configuration like this:

Host servername
     User ssh-username
     Hostname servername.server.tld
     Port 22
     IdentityFile ~/.ssh/id_ecdsa_sk

This now means that you can just type ssh servername and it’ll automatically use the right user, hostname, port and key.

…and without the hardware key?

If you don’t have your hardware key plugged in to you machine when you try to connect, then you’ll get this error:

Using Additional Configuration Files with Laravel & Lumen

2020-05-21T00:00:00-00:00

If you’ve got a growing Lumen application or perhaps you just want clear separation of configuration - it can be beneficial to add more configuration files to the config/ directory.

These files are pretty simple in nature, if I was creating one to configure settings for users, I could create a file called config/user.php containing:

return [
	'prefix' => 'user_',
	'table' => 'users',
	'permissions' => [
		CanPost::class,
		CanDelete::class,
		CanLogin::class,
	],
];

Laravel will load this file when accessed by default but Lumen on the other hand doesn’t. To get access to it via the config() helper, you’ll need to add the following line:

$app->configure(‘user’);

To your bootstrap/app.phpfile which will tell Lumen that there is an additional configuration file to use. After this has been done, you’ll be able to use the config() helper like normal. For example, to get the permissions we configured above:

$permissions = config(‘user.permissions');

You can read more about configuration files on the Laravel website.

Git reflog can help you restore lost work

2019-10-26T18:38:00-00:00

A lot of the software developers I’ve interacted with hold a basic working
knowledge of git which I’ve always found surprising as a lot of pieces
sound a lot scarier to hear than they are to use. An example of this would
be git rebasing - people immediately seem to worry about this even though at
the end of the day its a relatively simple concept - you have a stack of
commits and you want to trim them down.

Another one is the git reflog - there are times in your career when you
may need to use the reflog to rescue yourself.

In an existing profile, type git reflog and you’ll be presented with a
view similar to this:

If you want to make it a little easier to work out when each change was
made, then tag on --date=iso and run that instead:

git reflog --date=iso

and you’ll get the same output as above but with time and dates:

What this shows you is all the activity that has taken place in the
repository. It’ll show you merges, rebases, pulls, commits and more. It
will also contain all changes in your repo that were not
committed/staged as well - meaning you can recover accidental deletions
and rectify other mistakes. For each of the HEAD@{N} lines, you can
check each of those lines out, by using:

git checkout HEAD@{9}

If you want to really see everything that has happened to your
repository then you can actually pass an extra flag to git reflog
like this:

git reflog show --all

You can find more information and some extra reading at the
following sites:

I’ve hope you found this post informative enough to be confident in
exploring everything git can offer you.

How-to set downloaded filename for S3 pre-signed URLs in PHP

2019-10-01T20:54:00-00:00

If you use S3 to store files then the chances are at some point you’ll be
letting users download these files themselves. If you don’t want to fetch the
file yourself and you want to send the user directly to S3 to download it - you
can do this securely using pre-signed URLs. In short, a temporary URL is
generated to allow the user access to the file and it has a timed expiry.

If you’re normalising and storing your files with something like this structure:

file_id     file_name       file_type       aws_path

1022        sample          pdf             uploaded/documents/1022.pdf
1023        budget          xlsx            uploaded/documents/1023.xlsx

If you were sending the user to S3 directly, they’d be given a file to download
of 1022.pdf - hardly an ideal filename. You can modifying the filename for the
user when you create the pre-signed URL, like so:

$s3Client = new Aws\S3\S3Client([
    'profile' => 'default',
    'region' => 'us-east-2',
    'version' => '2006-03-01',
]);

$cmd = $s3Client->getCommand('GetObject', [
    'Bucket' => 'my-bucket',
    'Key' => 'testKey',
    'ResponseContentDisposition' => 'attachment; filename=sample.pdf'
]);

$url = $s3Client->createPresignedRequest($cmd, '+20 minutes');

Header("Location: ".$url); exit();

Once you then forward the user to the give URL they will be given a download
with the correct file name.

How to Zip and Encrypt a file on a Mac

2019-05-11T18:35:00-00:00

I’d recently seen a few people recommending app store apps to handle zipping and encrypting files/directories on a Mac.

Open up a Terminal session and locate the directory you need and run:

zip -er secret-files.zip directory-of-files/things.txt

You’ll then be prompted to enter the password and confirm it.

Strong Customer Authentication (SCA)

2019-04-10T20:22:00-00:00

Later this year on September 14th a new European payment law will come into use. For consumers its a brilliant step forward. Essentially the new law requires two-factor auth when checking out which should cut down on card fraud.

If you’re a Stripe user, all you’ll need to do is implement 3D Secure 2. This will cover your requirements for the new law. I asked Stripe on Twitter to confirm this (which they helpfully did):

Yep—you'll want to confirm that your integration is one that supports 3D Secure 2. https://t.co/bnFH0XmU3a can help get you started on this!
— Stripe (@stripe) April 9, 2019

I’ll be sticking with Stripe going forward, purely because they will handle most of this for me. You can read more information about what’s needed over on the Stripe docs for Strong Customer Authentication.

PHP 5.6 Vagrant Box for Legacy Projects

2019-03-14T19:30:00-00:00

I’ve never one to say there shouldn’t be any legacy projects. I’m sure plenty of
developers who would like to change the definition of a legacy project to suit
this weeks best practices but in all honestly there are legacy applications out
there generating hundreds and thousands of pounds for many companies. That in
itself should be respected.

Legacy has become a dirty word somehow. It shouldn’t be. We all have legacy
projects to work on.

Joe’s latest project - Homestead Fifty Six
has various price points to support the project. Joe has structured these in a
very fair manner I believe. For a project which will clearly save developers a
lot of time (I’ve recently had to create two Vagrant PHP5.6 boxes and each easily
took over a few hours).

Firefox Send

2019-03-13T21:10:00-00:00

Mozilla unveiled a new product this week called Firefox Send. It allows you to securely transfer files to friends, family and colleagues and anyone else around the world in a few clicks.

I used to use transfer.sh to do this on a self-hosted instance but going forward I’ll be using this.

Nginx acquired by F5 Networks

2019-03-12T20:02:00-00:00

A piece of news that seems to have been under the radar this week is that F5 have acquired nginx. I’ve been using nginx for a long time now along with countless others in the developer/devops community - I’m hoping that this doesn’t mean too many changes going forward to what is an excellent product.

You can read the press announcement here.

Automatically Start Laravel Horizon with Vessel

2019-02-15T20:50:00-00:00

If you’re a more advanced Laravel user there is a fair chance you will be using
Laravel Horizon to run your queue
configurations. If you’re using the excellent Vessel via Shipping Docker/Chris Fidao
then it can be annoying to have to drop into a container and manually start
Horizon.

Create a new file in your project named start_horizon.sh in the docker/app
directory, enter the contents of the file which should be:

#!/usr/bin/env bash
/usr/bin/php /var/www/html/artisan horizon

Save that file, and chmod +x start_horizon.sh. Next, open the file
supervisord.conf in the same directory and add:

[program:horizon]
command="/var/www/html/docker/app/start_horizon.sh"
directory=/var/www/html/
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

near the bottom of the file. Save the file and then rebuild your Vessel image,
using ./vessel build. After you restart your project with vessel up you
will have a running Horizon instance that you no longer manually need to start.

Laravel Nova - Trimmed Field Pattern

2019-02-06T21:00:00-00:00

In a Laravel Nova based project I had a list of domains that
sometimes grew to take up most of the row it was in. I discovered the following
pattern to show a limited field on the index view and a full field on the edit
view using a combination of onlyOnIndex() and hideFromIndex():

Text::make('Domains', 'additional_domains')
    ->onlyOnIndex()
    ->displayUsing(function ($value) {
        return str_limit($value, '100', '...');
    })
    ->withMeta(['extraAttributes' => [
        'readonly' => true
    ]]),

and then a second field that is hidden from the index using hideFromIndex():

Text::make('SSL Domains', 'ssl_additional_domains')
   ->hideFromIndex()
   ->withMeta(['extraAttributes' => [
       'readonly' => true
   ]]),

The fields are automatically populated by the system and are shown read-only to
the user so this really helped to tidy up the UI. Hopefully this helps you out
if you need to limit a field you’re displaying when using Laravel Nova.

Exa - A replacement for 'ls'

2019-02-06T21:00:00-00:00

Exa is a drop in replacement for your Mac’s standard tool ‘ls’. It provides a wider screen listing over ls along with useful extras like Git information. In the screenshot below you can see a N depicting new file within the posts directory.

It features really useful colour coding that takes a short while to get used to but after that allow quick identifying of files and runs on a whole host of different operating systems too.

I’m that much of a fan for exa, I’ve added it to my recommendations page with other great software.

Laravel Forge - LetsEncrypt Certificates Failing to Renew

2019-02-06T21:00:00-00:00

When LetsEncrypt arrived a few years ago, it definitely had one of largest
impacts on the web in living memory. Suddenly everyone could have SSL for their
website with minimal effort.

Over time, the way that systems like LetsEncrypt have to change the way they
work for operational and security reasons. One of these recent changes means
that your existing issued certificates may cease to renew. The reason for this
is because the TLS-SNI-01 challenge has been revoked.

You can see plenty of posts around the web discussing the change and the fix isn’t too painful to implement.

For Laravel Forge and other Ubuntu 16 compatible systems, you can run the
following commands to upgrade your version of certbot and get back to issuing
your certificates:

sudo apt-get update
apt-get install software-properties-common
add-apt-repository universe
add-apt-repository ppa:certbot/certbot
apt-get update
apt-get install certbot python3-certbot-nginx

Then run certbot and it’ll prompt you with a list of your sites to update:

From there you’ll be able to issue certificates as normal and auto-renewals
should succeed.

Thoughts about Domain Registration

2019-01-09T20:00:00-00:00

The following five points are issues that I believe should push people into taking their domain names into their own control.

Refusal to transfer

The first problem that can occur is when you decide to change your developers, if they have your domain names then you will have to negotiate the release of them to you.

Critical services are out of your control

Along with your domain name there is likely business critical services like email that are attached to your domain. You should know exactly where these services are being provided from and how to administer/control these. If you have an employee that goes rouge and you need to disable email services in an emergency - you need to be prepared for such an event.

Developer could go into administration or cease working

Along with the previous point there are plenty of reasons why you may suddenly have to access your domain name. If your hosting company goes into administration and you need to transfer your website immediately - without the domain being in your control neither you or a new developer can get your site/application setup without access.

You should know/understand where your domain name(s) are

Along with your domain name, there are other important pieces of your digital infrastructure that you should be aware of. Who provides your domain registration, who provides your DNS services? Who provides your organisations email services? These are all important things that you should know but also be aware of where you can administer them.

You could be paying over the odds for renewals

I’ve seen many domain registration companies who over-charge for the domains they register. It’s a good profit earner for smaller companies. Over the long term, this may be costing you a considerable amount of money. You can find the lowest fees by checking either Google domains or Namecheap. Both charge the lowest possible fee.

If it goes wrong

If you’ve got a .co.uk domain then you can raise a dispute with Nominet - for other TLDs then you’re going to have
to attempt to get in contact with the person who registered the domain.

Goals for 2019

2019-01-01T09:00:27-00:00

Each year I usually write myself a set of goals that I’d like to aim for the
next year. I think a lot of people try and set new year resolutions but I never
felt too compelled to do that (or to keep at them).

Finish financial changes
Lose weight and go back down two waist sizes
Read at least 14 books over the calendar year
See a sunrise/sunset with my partner
Learn and heavily integrate a new tech, either Vue or React.
Go to two music concerts (George Ezra booked for March)
Launch a new SaaS app
Two full (at least 10 day) vacations
Hip replacement and recovery

I think these are all hopefully achievable and within my control (apart from
hip replacement but my doctors and surgeon is fairly certain that’ll be very
early 2019).

I’ll come back through the year and update this list and mark each item
completed.

14 Books for 2019

2018-12-30T21:07:20-00:00

One of my goals for 2019 is to read at least 14 books. This post tracks the progress of that goal.

Current & Completed

How to be a Footballer by Peter Crouch
Day the World Came to Town: 9/11 in Gander, Newfoundland by Jim DeFede
Zucked by Roger McNamee
The Shortest History of Germany by James Hawes
Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration by Ed Catmull

Planned

The following list of books are ones that I’m planning to read in the future. You
can also check out my profile on Goodreads where it’s likely to be a bit more
up to date than this list.

Customising Exports for the 'Laravel Nova Excel' Package

2018-12-27T13:42:50-00:00

I recently worked on a project that needed an export for an Orders view within a Laravel Nova panel. If you have simple requirements, for example, exporting all fields with titles – that is relatively simple to produce. However, I needed to update fields and merge others with some other custom changes.

In short, I moved the export logic into its own Laravel Nova action called ExportOrders. and added this into the Nova Resource actions like so:

namespace App\Nova;

use App\Nova\Actions\ExportOrders;

class Order extends Resource
{
    // snipped..
    /**
     * Get the actions available for the resource.
     *
     * @param  \Illuminate\Http\Request  $request
     * @return array
     */
    public function actions(Request $request)
    {
        return [
            (new ExportOrders())
        ];
    }
}

Next, I created the above-named file at app/Nova/Actions/ExportOrders.php to define the headings and map the fields to their corresponding export fields:

namespace App\Nova\Actions;

use App\Models\Order;
use Maatwebsite\Excel\Concerns\WithHeadings;
use Maatwebsite\Excel\Concerns\WithMapping;
use Maatwebsite\LaravelNovaExcel\Actions\DownloadExcel;

class ExportOrders extends DownloadExcel implements WithMapping, WithHeadings
{
    /**
     * @return array
     */
    public function headings(): array
    {
        return [
            'internal_order_id',
            'created_at',
            'stripe_transaction_id',
            'buyer_name',
            'buyer_email',
            'billing_address',
            'delivery_name',
            'delivery_address',
            'giftcard_amount',
            'total_paid',
            'message',
            'status'
        ];
    }

    /**
     * @param $order
     *
     * @return array
     */
    public function map($order): array
    {
        return [
            $order->id,
            $order->created_at,
            $order->transaction_id,
            $order->buyer_name,
            $order->buyer_email,
            $order->billing_address,
            $order->recipient_name,
            $order->delivery_address,
            $order->amount,
            $order->total,
            $order->notes,
            $order->status
        ];
    }
}

Initially I’d added more functionality within this class to merge fields and add custom export but in the end, I decided to move a lot of these field customisations into Eloquent accessors instead. This meant that when they were accessed in the Nova panel, exports or anywhere else they became consistent.

Easy Screen Capture on macOS with Giphy Capture

2018-09-11T20:16:45-00:00

I recently asked on Twitter on what people used for screen capturing. I
actually wanted terminal recording to highlight some commands being ran for a
talk about serverless. I then came across the brilliant
Giphy Capture app and it’s so
simple to use and produces perfect results every time.

If you need to record your screen for whatever reason on your Mac then I can
highly recommend it.

Thoughts about Mastodon

2018-09-08T11:51:23-00:00

I’ve seen quite a few people posting to Twitter recently that they’ve tried or are actively trying out Mastodon as an alternative to Twitter. This gave me the chance to actually have a few good conversations on the subject and I decided to note down some observations from what I’ve seen. I’m going to preface this and expect that you have an understanding of what Mastodon is – if you don’t then read this ‘Meet Mastodon’ by Tiffany Brown.

The Problem

Twitter (along with most social networks) can be a very toxic place. Recently an effort has been made to counteract this but I don’t think that has been anywhere near successful. The toxicity problem isn’t just confined to Twitter though – you can head over to Facebook, Reddit and YouTube among others and see the same on those platforms. I’d even go as far as saying the same toxicity is present in the developer community as well which for a long time felt like it didn’t have that problem. I think as development has become more commonplace it’s inherently gained some of the unpleasant aspects of society and that transcends every method of communication.

Mastodon prides itself on giving users better control of protecting themselves within the social environment. Users can not only block other users of the same instance – they can block entire instances. There are additional controls built in but like anything it isn’t, can’t and won’t ever be foolproof. I think headlines like Motherboards ‘Mastodon is like twitter without the nazis’ don’t help the situation and give a false sense of safety – each and every platform suffers from toxicity. Mastodon is still small enough that it hasn’t attracted the bad crowds yet (I have absolutely no doubts that there are instances already around that are dedicated to horrendous things).

It’s a little like when folks used to say Mac’s wouldn’t get viruses because they weren’t targeted – the market share was too small for hackers to care. Once the Mac and iOS platforms both got larger they became attack targets fairly quickly. Mastodon will be no different to those.

Prominent Trying Out

I’ve seen David Chartier, Eric Barnes, Ian Landsman among others testing out Mastodon recently.

Brand Use

Part of the conversation I had with Eric was that Mastodon just doesn’t suit a brand or personal brand presence anywhere near as well as Twitter. If you instruct someone to follow @ericlbarnes on Twitter then it is fairly easy and failsafe to do so. Mastodon suffers the same problem as ICQ did. In the days of ICQ, you had to remember a long number which was your identifier – mine was 32646413. Not easy to remember, but still slightly easier than Mastodon to find the right person.

The next issue along is verification and proving that someone who says they are X really is that person. There is no built-in mechanism to verify that a person is exactly who they say they are within Mastodon. Usernames on Mastodon look a lot like a combination or a twitter username and email address. For example, these three could be Elon Musk:

@elon@elonmusk.com

@elon@tesla.com

@elon@spacex.com

There is zero chance of you verifying which (if any) is the real Elon Musk. Twitter solved this by adding a blue mark next to profiles that they had independently verified but there is no such process with Mastodon.

So back to Eric, there could be an @ericlbarnes located on the more popular mastodon.social instance, or mastodon.cloud or even ericlbarnes.com – there is no method to verify which actually is Eric so there isn’t a canonical place for people to find brand Eric.

The closest that people have come to be able to do this is writing a page on their website detailing which user and instance they are. Straight away, you’re making people jump through hoops to do something that should be quick and easy. That makes it a much tougher sell to someone to use Mastodon as a brand communication tool.

Smaller Communities & Closer Discourse

I’ve used Mastodon for quite a while and the thing that still sticks with me to this day is that it really feels like there is a value of conversation. I think part of that comes from it being a smaller environment but the where Twitter seems to have a post-and-forget feel – Mastodon has a conversation feel.

A lot of people including Eric think that Mastodon will die out soonish with the likes of Myspace, Digg et all but I think that its resonation with getting back to the conversation will make it stick for a lot of people.

Freeze - A Mac Client for Amazon Glacier

2018-08-26T16:23:40-00:00

I tested a couple of recommendations for AWS’s Glacier and settled upon
‘Freeze‘. It’s reasonably
priced, great to use and has every feature you’ll need.

Pushing Ottomatik Webhooks to Slack via Apache IBM Cloud Functions OpenWhisk

2018-07-21T15:57:28-00:00

The backup service Ottomatik implemented webhooks last year (at my request, ha!) to send success or failure payloads to a URL of your choice. As Ottomatik doesn’t yet provide a Slack integration. Most teams around the world use Slack to pipe notifications into their channels and it’s a pretty good way of doing things.

You’re going to need to set up a webhook for Slack which you can do on their site. To make things easier, you can use serverless project to make dealing with serverless providers easier. Follow the install steps listed on serverless website to get started.

We’ll use Guzzle to send our API call to Slack. We’ll put all of the code that executes our function in a file named handler.php.

You can clone this repository ssx/openwhisk-ottomatik-to-slack as a quick start.

Update the URL for the Slack webhook and we’ll then push our function up.

sls deploy

Here’s everything in one go:

which will return a webhook for you to use in Ottomatik:

If everything went to plan, you’ll now get near real time updates on how your backups are performing.

Comments, idea and thoughts welcome over on Github.

Ubuntu Zesty Fails to 'apt-get update'

2018-05-14T17:23:33-00:00

Ubuntu Zesty no longer is a supported version of Ubuntu, you’ll need to
update apt/sources.list to this:

deb http://old-releases.ubuntu.com/ubuntu/ zesty main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ zesty-updates main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ zesty-security main restricted universe multiverse

deb http://old-releases.ubuntu.com/ubuntu/ zesty-proposed main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ zesty-backports main restricted universe multiverse

Run apt-get clean all after updating that file and you’ll be able to
upgrade as normal.

Clear Bash History

2018-04-19T14:51:45-00:00

Quick way to clear bash/shell history:

cat /dev/null &gt; ~/.bash_history && history -c && exit

Deleting Immutable Files on Linux

2018-04-15T13:08:15-00:00

If you have a file such as php-fpm.cgi that won’t delete no matter what
you try, it’s likely immutable. You can remove the immutable flag using:

chattr -i file.name

Easier Git Aliases

2018-04-14T13:01:28-00:00

I recently came across the git-legit project which creates a few easier to
understand aliases for newcomers to Git. Head over to http://www.git-legit.org
to read and install.

Install older Android SDK Version

2018-04-01T20:49:58-00:00

For a Titanium project I recently needed to install an older version of the Android SDK tools (build tools API 24) to build an old project.

Find the path to sdkmanager and then run this command:

sdkmanager --install "platforms;android-24"

How to De-Register iMessage

2018-03-25T22:15:15-00:00

This is quick and easy, head over to https://selfsolve.apple.com/deregister-imessage/
and fill in your mobile number. It will then SMS a code which you need to
input into the site and then deactivate iMessage for that number.

List All Your Digital Ocean Servers IPv4 addresses

2018-03-07T16:27:15-00:00

List all Digital Ocean servers IPv4 addresses using doctl homebrew package:

doctl compute droplet list --format PublicIPv4 > ips.txt

AWS S3 Recursively List Bucket Sizes

2018-03-07T16:23:36-00:00

Recursively list sizes of your AWS buckets using s3cmd:

scott@atlas.local  ~ s3cmd du -H -r s3://                                                                                                                                                                        16:19:22

3.81335246842G 51 objects s3://ssx-general-backups/

52.0762854349G 52 objects s3://ssx-archived-client-websites/

189.562103216G 565 objects s3://ssx-backups-production/

664.248046875M 1 objects s3://ssx-backups-git/

7.44284629263G 74 objects s3://ssx-backups-servers-shared/

238.997132301M 3579 objects s3://ssx-images/

253.776662782G Total

Simple IP Reverse & Info

2018-02-13T10:38:11-00:00

Using ipinfo.io, you can quickly grab some IP data for the current machine:

curl ipinfo.io

This will return you something like:

scott@atlas.local ~ curl ipinfo.io                                   
 {
   "ip": "195.62.123.123",
   "city": "Stafford",
   "region": "England",
   "country": "GB",
   "loc": "52.8052,-2.1164",
   "org": "AS6717 Internet Central Limited",
   "postal": "ST16"
 }

Apache Set Server Tokens Off

2018-02-11T13:50:11-00:00

ServerSignature Off
ServerTokens Prod

Apache Set Expires Header for Images/CSS

2018-02-10T10:49:14-00:00

<IfModule mod_expires.c>
ExpiresActive on
ExpiresDefault "access plus 30 seconds"
ExpiresByType text/html "access plus 15 days"
ExpiresByType image/gif "access plus 1 months"
ExpiresByType image/jpg "access plus 1 months"
ExpiresByType image/jpeg "access plus 1 months"
ExpiresByType image/png "access plus 1 months"
ExpiresByType text/js "access plus 1 months"
ExpiresByType text/javascript "access plus 1 months"
</IfModule>

Apache Redirect for Canonical Domain

2018-02-09T18:00:25-00:00

RewriteEngine on
RewriteCond %{HTTP_HOST} !^www.example.com$
RewriteRule (.*) https://www.example.com$1 [R=301,L]
RewriteCond %{HTTPS} !=on
RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]

SPF Record for No Mail Sending Domain

2018-02-07T21:46:27-00:00

"v=spf1 -all"

Namecheap SSL Certificate Combining

2018-02-05T14:28:42-00:00

cat example_com.crt COMODORSADomainValidationSecureServerCA.crt COMODORSAAddTrustCA.crt AddTrustExternalCARoot.crt >> ssl-bundle.crt

Laravel Validation Rules

2017-09-10T14:47:59-00:00

I’ve noticed over the past few months my usage of the Laravel validation
engine has increased a lot. I’ve even used it in some non-Laravel
applications just because I’m a fan of the interface. I had the idea of
sharing some of the validation rules with the community but there wasn’t
too many people doing the same.

I created a new project to address this, called
Laravel Validation Rules. You
can follow @laravelvr on Twitter for project
updates, check out the current rules at laravel-validation-rules.github.io to
see if any of the rules can help you in your project.

If you have your own rule that you think the community would benefit from
having, please submit it.

Quick Way to Set Telegram Webhook

2017-09-09T14:57:08-00:00

I needed a small tool recently to set Telegram webhooks quickly over a few projects. You usually just run a curl command to do this but the format is something you have to remember – this is a lot simpler. Just head over to https://telegram-set-webhook.tools.dor.ky/ to use it.

If you’d like to verify the source or contribute then you can find the project on Github at https://github.com/ssx/telegram-set-webhook.

Docker Based Torrent Client

2017-09-04T19:44:23-00:00

Easy way to run rtorrent in Docker is using Docktorrent like this:

docker run -it \
    -p 81:80 -p 45566:45566 -p 9527:9527/udp \
    --dns 8.8.8.8 \
    -v /rtorrent:/rtorrent \
    -e UPLOAD_RATE=1024 \
    kfei/docktorrent

Writing a Better README.md

2017-08-26T13:23:16-00:00

A while ago I remember seeing a tweet from Ryan Florence about docuemntation:

I think the biggest mistake we all make in docs and tutorials:
Not explaining *first* what the goal, or use-case, or desired result is.

— Ryan Florence @ 🐙🤖 (@ryanflorence) June 19, 2017

Last week I’d seen that Kent C Dodds had a further point on it, which I liked:

Here's the order I use in my projects:

Problem
Solution
Installation
Usage
Inspiration
Related
Thanks
Licensehttps://t.co/nCBk52VOiu

— Kent C. Dodds (@kentcdodds) June 20, 2017

So going forward I’m planning to use that as a template along with going over some existing projects and modifying those. I think the extra information contained within the problem and solution areas could really help others who are trying to solve their own problem.

Magento Redirection after Store URL Changed

2017-08-24T12:55:49-00:00

I recently ran into an issue where a Magento site that had been duplicated for development kept redirecting to the original site URL. Turned out that the culprit was a setting called SSL_OFFLOADED which needed to be turned off.

Tip: Telephone Preference Service

2017-08-22T19:13:32-00:00

If you’re in the UK, you can easily stop a lot of marketing calls by signing up for the Telephone Preference Service. It’s extremely simple to do, send a text message to 85095 with ‘TPS email@address.com” – don’t forget to remove the quotes and add your email address in.

You can find full instructions over on the TPS website.

Restore Telnet in Mac OS High Sierra 10.13

2017-08-17T15:40:19-00:00

Telnet has been removed from Mac OS High Sierra, but can still be useful for developers to check ports are open etc.

If you need to enable it, you can do so with Homebrew:

brew tap theeternalsw0rd/telnet                                                                                                                                                                                                                   
brew install telnet

Update (10/10/17): I received an email from David Waitzman in which he notes you can get telnet from an old Time Machine backup and then copy it into /usr/local/bin – I’d recommend that options too if you have a backup.

Update (25/11/17): I received a second email from Michael Bazik which details how you can install the telnet client directly from GNU provided tarballs:

curl http://ftp.gnu.org/gnu/inetutils/inetutils-1.9.4.tar.gz -o inetutils-1.9.4.tar.gz
tar xvzf inetutils-1.9.4.tar.gz
cd inetutils-1.9.4
./configure
make
sudo make install

Update (05/01/18): Jonny Bradley emailed in to address an issue with installing non homebrew telnet. If you encounter the error:

ping: Lacking privilege for raw socket.

then you’ll need to chmod ping to allow it permission to run with:

sudo chmod a+s /usr/local/bin/ping

Update (08/02/18): Matthew Ward got in touch to say that you can compile telnet without all of the other gunk that still comes with Mac OS, using the following:

# out of the "inet-utils" bundle from GNU, the only utilities it provides
# that Mac doesn't "natively" are:
#
#   ftp     the ftp client
#   telnet  the telnet client
#
# you don't need to build the whole set. use the following "configure"
# command line:

./configure --disable-servers --disable-dnsdomainname \
--disable-hostname --disable-ping --disable-ping6 --disable-rcp \
--disable-rexec --disable-rlogin --disable-rsh --disable-logger \
--disable-talk --disable-tftp --disable-whois --disable-ifconfig \
--disable-traceroute

MacOS: Add Message to Lock Screen

2017-06-22T19:38:14-00:00

This is a really good idea about adding a message to your lock screen on MacOS with a note on how to contact you in the event your Mac is lost.

Testing Laravel Applications with CircleCI & Dusk

2017-05-17T21:01:21-00:00

I recently started a new project and wanted to give Laravel’s new test framework Dusk a try. Locally, this was quick to do and it worked first time – however when I then connected up CircleCI I needed to do a little extra work to get up and running. In short, the version of Google Chrome needs updating to a newer version and then things should be running well.

Firstly, make sure you’re up and running locally and your tests are passing (that’s half the work done, right?). You should have output similar to:

scott@inex.local / sample-app / master ✔ : php artisan dusk
PHPUnit 5.7.19 by Sebastian Bergmann and contributors.

...                                                               3 / 3 (100%)

Time: 15.18 seconds, Memory: 16.00MB

OK (3 tests, 3 assertions)

So, we’re all good at this point. Next, get your circle.yml file looking something like the following:

machine:
  pre:
    - sudo apt-get update; USE_PRECOMPILE=true sudo -E circleci-install php 7.1.0
  timezone:
    Europe/London
  php:
    version: 7.1.0
  environment:
    APP_ENV: "testing"
  hosts:
      testing: 127.0.0.1

general:
  artifacts:
    - "tests/Browser/screenshots"
    - "tests/Browser/console"

dependencies:
  pre:
    - rm /opt/circleci/php/$(phpenv global)/etc/conf.d/xdebug.ini

  override:
    - composer install
    - wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
    - sudo sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
    - sudo apt-get update
    - sudo apt-get install google-chrome-stable

test:
  pre:
    - "./vendor/laravel/dusk/bin/chromedriver-linux":
        background: true
    - cp .env.testing .env
    - cp .env.testing .env.dusk
    - "php artisan serve":
        background: true

  override:
    - mkdir -p $CIRCLE_TEST_REPORTS/phpunit
    - php artisan dusk --log-junit $CIRCLE_TEST_REPORTS/phpunit/junit.xml tests

This will make sure you’re using PHP 7.1, start the site running in the background and allow dusk to run. You’ll need to make sure that the APP_URL value in your .env.testing file is pointing to 127.0.0.1. In the example above, we’re making sure that CircleCI adds an entry to /etc/hosts for ‘testing’.

Providing your tests now work, you’ll have a big green bar in CircleCI.

Use 'net-alias' in Docker for Easy Round Robin Load Balancing

2017-05-17T20:36:05-00:00

Something I picked up on a few days ago was using Docker’s –net-alias command (or docker-compose equivalent) for cheap and easy round robin load balancing.

For example, start two elasticsearch containers with –net-alias search and from within the network, using the ‘search’ as the DNS name for connecting, you’d get 1 of the 2 servers at random.

Tool to Create Mastodon OAuth2 Applications in Your Browser

2017-04-19T20:35:17-00:00

At the moment, there isn’t a user friendly way of setting up new OAuth2 applications for use with Mastodon. You need to make an API request to do so. To help, I’ve written a small tool which allows you to do this securely in your browser.

You can use the tool at mastodon-create-app.ssx.tools or you can grab the source yourself from Github at ssx/mastodon-create-app.

If you have any ideas or feedback, please open an issue on the repo.

Github & Hide Files from Commit Diffs

2017-04-16T11:26:53-00:00

This is a really nice tip for tidying up git diff‘s on Github (I think it’ll work for Gitlab etc too), you can add the following line to .gitattributes file:

dist/* linguist-generated

and this will hide all files in the dist/ directory from the commit diff.

Appcelerator Titanium Starting an Intent/Activity Not Firing

2017-04-15T20:02:54-00:00

If you’re a user of intents with Android and Appcelerator’s Titanium product, you may have instances where your intents don’t fire as you expect. A lot of sample code around the web for starting intents looks something like this:

var intent = Ti.Android.createIntent({
    action: Ti.Android.ACTION_MAIN
});
intent.addCategory(Ti.Android.CATEGORY_LAUNCHER);
Ti.Android.currentActivity.startActivity(intent);

The problem with arises when sometimes Ti.Android.currentActivity doesn’t correctly hold the current activity. When you then attempt to start an intent from the current activity, it does nothing. The way around this is to call startActivity from the current view, like this:

$.nameOfCurrentView.startActivity(intent);

Which will fire the new intent every time.

Continuous Integration Testing an Appcelerator Titanium Mobile Application

2017-04-15T09:10:27-00:00

I’ve recently been looking at unit testing some isolated components for a mobile application I’m working on. Part of the application is a calculations engine which I wanted to test against a known set of sample data and results.

There have been many attempts through the community at putting widely accepted testing frameworks throughout the last few years, but none felt too akin to what I wanted or how I wanted it to work. Most of the testing frameworks out there such as timocha run within the application via a simulator and this was something I wanted to avoid.

For version control I use both Github for personal projects and then Gitlab at work and for my larger personal projects. I wanted a solution that could run within Gitlab’s CI runner – which I use Docker.

I began by creating a Dockerfile which included the things needed to build an application. There were a few road blocks in the way. The image needed to have node, npm, Oracle Java 8, android-idk-linux and a few npm packages such as appcelerator, titanium, jasmine, tisdk.

Firstly, the appc command will require a username and password to login to to Appcelerator’s platform to fetch an SDK and produce a build – this wasn’t ideal. The appc tool won’t run under the root user (for good reason) which also meant employing runuser a few times to get commands to run.

A few very helpful conversations with Adam Paxton highlighted the useful tisdk npm package which will fetch and install SDKs and the open source ti NPM package can perform the open source version of appc which includes producing a build. You’ll also need to install some gcc libraries as well (they’re in the Dockerfile).

The producing a build part is needed with tiunit so that you can correctly include any alloy resources that you need (in my case, I had used alloy/moment in a few places which I ended up mocking instead. For reference, you can build your application using (adjust for platform you’re using):

appc run -p android -b

In the end, I managed to get my tiunit tests running, I’ll follow up with a sample application and test case for you to follow along with.

You can use the image from Dockerhub at hellossx/appcelerator-ci which is built automatically from the Github repo ssx/docker-appcelerator-ci. I used the following .gitlab-ci.yml configuration to have it run my tests:

image: hellossx/appcelerator-ci

variables:
  JAVA_HOME: "/usr/lib/jvm/java-8-oracle/"
  ANDROID_HOME: "/usr/local/android/android-sdk-linux"

cache:
  key: "costings_app"
  untracked: true

stages:
  - test

npm_tasks:
  stage: test
  script:
    - npm install

run_jasmine_tests:
  stage: test
  script:
    - tisdk install 6.0.3.GA
    - ti build -p android -b
    - npm test

If you prefer that as a gist, you can view it here.

Notes:

Hyperloop is a no-no using this method, without introducing your Appcelerator username/password into the scripts.
Introducing Appcelerator username/passwords would be trivial using Gitlab’s secret variables within their CI.

Magento 2 Interceptors

2017-03-28T17:04:40-00:00

A colleague at work told me about the way you can use interceptors in Magento 2 to wrap up before/after functionality of existing classes. This is a pretty good overview of how that works written by Jisse Reitsma.

Zapier's CLI Tool for Creating Zaps

2017-03-27T18:20:04-00:00

This is a pretty neat new feature from Zapier to allow developers to build the little modules that connect apps together. They’ve released a CLI tool which will scaffold an app. You can take a look over on their developer blog in this post.

Removing Vue From Default Laravel Installations

2017-03-26T17:56:28-00:00

Matt Stauffer has a great post on how to remove Vue from default Laravel installations.

App Performance Low Hanging Fruit

2017-03-25T12:38:49-00:00

One of my favourite developers, Andrew Del Prete has written a new mini blog series on quick wins for performance and low hanging fruit changes that can be made to give you easy wins.

You can find the first (and other five parts) over on the learnperf.com blog.

Overview of Apple's iCloud Security & Encryption Features

2017-03-24T12:33:13-00:00

A talk given at Blackhat conference this year explain how Apple ensure that their customer data is safe. You can download a PDF of the slides here.

All About Google's AMP

2017-03-23T12:28:59-00:00

This is a really good overview of Google’s AMP project and how it came to be.

Breaking down AMP—the format, rules, and cache—from @adactio. https://t.co/rX6AWoYJ8U

— A Book Apart (@abookapart) March 23, 2017

Magento SQL Profiler

2017-03-22T12:27:30-00:00

I’d seen a tweet to Ben Marks this week which detailed an SQL profiler for Magento 1.

How Tube Stations Got Their Name

2017-03-21T12:26:17-00:00

Pretty interesting post on how some of the more obscurely named stations got their names.

Android Install Older Version of Build Tools

2017-03-20T12:56:23-00:00

If you need to install an older version of Android’s build tools, you can do so using the below. Search for the SDK version you want:

$: android list sdk --all | grep -I tools

Which will then give you an indexed list of SDKs:

1- Android SDK Tools, revision 25.2.5
   2- Android SDK Platform-tools, revision 25.0.4
   3- Android SDK Build-tools, revision 25.0.2
   4- Android SDK Build-tools, revision 25.0.1
   5- Android SDK Build-tools, revision 25
   6- Android SDK Build-tools, revision 24.0.3
   7- Android SDK Build-tools, revision 24.0.2
   8- Android SDK Build-tools, revision 24.0.1
   9- Android SDK Build-tools, revision 24
  10- Android SDK Build-tools, revision 23.0.3
  11- Android SDK Build-tools, revision 23.0.2
  12- Android SDK Build-tools, revision 23.0.1
  13- Android SDK Build-tools, revision 23 (Obsolete)
  14- Android SDK Build-tools, revision 22.0.1
  15- Android SDK Build-tools, revision 22 (Obsolete)
  16- Android SDK Build-tools, revision 21.1.2
  17- Android SDK Build-tools, revision 21.1.1 (Obsolete)
  18- Android SDK Build-tools, revision 21.1 (Obsolete)
  19- Android SDK Build-tools, revision 21.0.2 (Obsolete)
  20- Android SDK Build-tools, revision 21.0.1 (Obsolete)
  21- Android SDK Build-tools, revision 21 (Obsolete)
  22- Android SDK Build-tools, revision 20
  23- Android SDK Build-tools, revision 19.1
  24- Android SDK Build-tools, revision 19.0.3 (Obsolete)
  25- Android SDK Build-tools, revision 19.0.2 (Obsolete)
  26- Android SDK Build-tools, revision 19.0.1 (Obsolete)
  27- Android SDK Build-tools, revision 19 (Obsolete)
  28- Android SDK Build-tools, revision 18.1.1 (Obsolete)
  29- Android SDK Build-tools, revision 18.1 (Obsolete)
  30- Android SDK Build-tools, revision 18.0.1 (Obsolete)
  31- Android SDK Build-tools, revision 17 (Obsolete)

and then you can install a specific package with the ID:

$: android update sdk -u -a -t 6

This blog post helped to discover the way to accomplish this.

Magento 2 Glossary

2017-03-19T10:57:17-00:00

Check #magento2 glossary – https://t.co/e0mSvIpbvg and give us feedback.

— Max (@maksek_ua) March 19, 2017

Pro Tip: Find All HP Printers on Your Local Network

2017-03-18T09:08:18-00:00

Using nmap (installable with homebrew) you can run the following to get the IP addresses of all the HP printers on your local network:

nmap -v -A 192.168.0.\* | grep 9100

Appcelerator Titanium – Set Build Tools Version

2017-03-17T09:06:43-00:00

If you have a newer version of build tools and need to fix the version currently in use, you can set the following:

appc ti config android.buildTools.selectedVersion 23.0.3

Git Tip: Display Contributions per User

2017-03-16T09:05:09-00:00

You can use the following to display committer name and their contribution counts:

git shortlog -sn

Base64 Encode & Decode on OSX

2017-03-15T09:01:02-00:00

On OSX you can quickly base64 encode:

base64 -in filename.txt > filename.b64

and decode files:

base64 -D -in filename.b64 > filename.txt

AWS Updates Quickstart for Magento 2

2017-03-14T08:58:45-00:00

Try out our updated Quick Start for Magento 2.1.5 on the AWS Cloud – with Amazon EFS & Amazon ElastiCache! https://t.co/Y81C9wAQQi pic.twitter.com/6b47n6TlRr

— Amazon Web Services (@awscloud) March 20, 2017

MET Police Using Indian Hacking Agency

2017-03-12T08:51:55-00:00

.@metpoliceuk used an Indian agency to hack activists and journalists in UK – then shredded the evidence. Just wow. https://t.co/g0Y4RD2SOn

— Matthew Butcher (@matthew1butcher) March 21, 2017

Quick Way of Remembering PHP Needle/Haystack Order

2017-03-10T08:48:16-00:00

Friendly reminder → in PHP, string functions are haystack/needle, array functions are needle/haystack (except array_filter).

— assertchris (@assertchris) March 21, 2017

We Made the Whole Company Serverless

2017-03-09T08:46:33-00:00

Interesting post about taking a server-less approach to running a cloud service.

Magento 2 Slow During Development

2017-03-08T08:45:00-00:00

If you've been complaining about M2 being slow during development: read this! Awesome summary! https://t.co/QiZ48WlVz9

— Anna Völkl (@rescueAnn) March 19, 2017

Quickly Generate SSL Key & CSR

2017-03-07T11:10:12-00:00

Quickly generate a new SSH keypair and then a CSR:

openssl genrsa -out private.key 4096
openssl req -new -key private.key -out private.csr

Quickly Start New Projects From Git Repo

2017-03-06T10:00:04-00:00

Found this piece of code a while ago which is useful if you want to start projects from a skeleton repo or similar starting point.

mkdir new-laravel-project
cd new-laravel-project
git init
git fetch --depth=1 -n https://github.com/laravel/laravel
git reset --hard $(git commit-tree FETCH_HEAD^{tree} -m "Initial Commit")

Installing Mailhog for Use with Laravel Valet

2017-03-05T15:07:03-00:00

Really decent article on how to use a homebrew installed Mailhog for Laravel Valet.

Add a Git alias to List Git Aliases

2017-03-03T17:42:44-00:00

git config --global alias.alias "config --get-regexp ^alias\."

OSX – Disable Gatekeeper to Allow Installation of Unsigned Apps

2017-03-02T17:42:07-00:00

Run this in your terminal to allow the option of selecting install from anywhere:

sudo spctl --master-disable

Use nano in Docker containers

2016-11-20T10:05:54-00:00

If for some reason you find yourself needing to use `nano` in a Docker container, you can set the terminal type and then you’ll be able to use it. Say for example you ran `bash` with the exec command, you can then run:

export TERM=xterm-256color

and it’ll allow you to use the terminal as normal for nano/clear etc.

teleconsole

2016-10-13T21:02:35-00:00

This is a nifty little app that will allow you to share a terminal console with others. It’s called teleconsole, check it out if you have a use case.

Set keyfile to use for SFTP

2016-09-16T09:28:02-00:00

If you want to specify the keyfile used for an SFTP connection, you can use the following:

sftp -i /path/to/private/keyfile user@machine1:/path/to/file user@machine2:/path/to/file

Vagrant SSL Error Fix

2016-09-12T11:02:38-00:00

Minor issue this morning where Vagrant wouldn’t fetch a box completely, turns out that the local cache looked corrupted and you would get this error:

SSL read: error:00000000:lib(0):func(0):reason(0), errno 60

you can force it to refetch entirely, by using the command:

vagrant box add box/name -c

then run your vagrant up as usual and it should work.

Auto Deploy Couscous.io Docs to Laravel Forge

2016-09-09T12:28:13-00:00

If you use couscous.io to generate documentation and Laravel Forge, you can set up auto deployment and publishing quite easily. Setup the site in Forge, making sure you have a composer.json and an empty public directory in your repo.

You can then use this deploy script to build, move and publish your changes automatically after you’ve added a webhook too:

Titanium Appcelerator: Install Specific SDK Version

2016-07-07T16:09:18-00:00

If you need to install a specific version of Titanium’s SDKs, use the following command:

appc ti sdk install 5.2.0.GA

Clone a git repo using a specific SSH private key

2016-05-22T10:00:48-00:00

If you wish to clone a repo using a set private key, you can use the following syntax to do just that and it won’t add the private key to your usual SSH settings:

ssh-agent bash -c 'ssh-add private.key; git clone git@server.com:user/repo.git'

Generating Public & Private Keys Using PHPSECLIB

2016-05-21T17:23:02-00:00

A few tasks that I’ve been working on this weekend have needed public/private key pairs. The very useful phpseclib library provides the tools that you need to generate these within your application.

// Get a new RSA object and set the type, hash and comment
$rsa = new RSA;
$rsa->setPublicKeyFormat(phpseclib\Crypt\RSA::PUBLIC_FORMAT_OPENSSH);
$rsa->setHash('sha256');
$rsa->setComment('user@seccheck');

// Actually generate the key, change 4096 for the desired number of bytes
$keys 		= $rsa->createKey(4096);
$keyPrivate	= $keys["privatekey"];
$keyPublic	= $keys["publickey"];

// Get the fingerprint for this key
$rsa->loadKey($keyPrivate); 
$keyFingerprint = $rsa->getPublicKeyFingerprint();

Embed Github files Gist Style

2016-05-20T08:27:28-00:00

If you’ve ever wanted to embed a Github file into your site, in the same style that Gist does, there’s a decent app over at http://gist-it.appspot.com that will do just that.

Searching the Internet for Devices

2016-05-18T08:43:46-00:00

This is a pretty nifty service I come across a few days ago. It’s called Shodan and searches then indexes internet connected devices around the world.

Using UUIDs in Laravel Projects

2016-05-17T09:02:00-00:00

There’s a good article on using UUIDs instead on integer IDs on on Medium by Steve Azzopardi. Head over to “Setting up UUIDs in Laravel” on Medium.

Chrome Casting Audio from Mac OSX

2016-05-16T20:17:11-00:00

If you’re a Mac OSX user and would like to send your audio over to a Chromecast device (ie, playing iTunes or Spotify) then there’s a fantastic project called soundcast which will allow you to do just that. You can find the project over on Github at https://github.com/andresgottlieb/soundcast.

Export Files from iTunes Playlist

2016-05-08T11:21:32-00:00

Handy tip: You can drag/drop files from an #iTunes playlist into a mount device on OSX. Useful for portable devices. #osx #mac

— Scott Wilcox (@ssx) May 8, 2016

The .well-known Directory on Webservers

2016-03-20T16:58:06-00:00

If you’ve often seen things looking for a directory called .well-known on your HTTP server access logs, you’d be like me if you were surprised to know there is actually quite a lot more to that directory name. It provides information about services and a whole bunch more. You can read a lot more on Mattias’s blog post about it.

Linux Server: Send Email on File Change

2016-03-14T18:00:45-00:00

This script will send an email when a file changes in the given path. Can’t remember where I picked this up from but it’s useful to have when you need to debug something that writes files.

#!/bin/sh
#
# Monitor file $1 for changes
# Send an alert emai to $2 if file $1 changes
# usage: file_change_mail_alert.sh /var/log/messages your.name@domain.com
#

if [ -z "$2" ]; then
echo "Usage: file_change_mail_alert.sh "
exit 1
fi

#if a inotifywait for this file is already running
if [ $(ps aux | grep inotifywait | grep -c "$1" ) -gt '0' ]; then
	echo "A process monitoring the file $1 is already running: $(ps aux | grep inotifywait | grep "$1" )";
	exit 1;
fi

#if inotifywait exists
type -P inotifywait &>/dev/null || { echo "Error: This script requires inotifywait(http://wiki.github.com/rvoicilas/inotify-tools/) .... apt-get install inotify-tools ... " >&2; exit 1; }

#if the file exists
# if [ -f $1 ]; then

echo "Monitoring file $1 for changes - sending alerts to $2"

while inotifywait -e modify -e attrib -e move -e delete $1 -o /root/audit.log; do
	sleep 1
	
	# changes="$(tail -n5 $1)"
	
	changes="$(cat /root/audit.log)"
		echo "The following change occurred in the file $1 : $changes" | mail -s "Change in $1" $2
		rm /root/audit.log
		touch /root/audit.log		
	done
else
	echo "Error: File $1 not found"
fi

Outbound FTP Issues with Virtualmin

2016-03-12T18:00:16-00:00

If you’re a Virtualmin user and use the iptables based firewall, you may have issues with outbound FTP working. The FTP connection can often be made, but then stalls at receiving data.

You can fix this by adding the nf_conntrack_ftp module to your machine. As root or sudo bash, create the following file:

/etc/modules-load.d/nf_conntrack_ftp.conf

and within that file put:

modprobe nf_conntrack_ftp

This will make sure that module gets loaded after each boot and fix your connection issue.

Making Local Development Faster with Caches for NPM & Composer (Part Two: Composer)

2016-03-11T18:00:41-00:00

If you’re a heavy PHP user, you’ll most likely be using packagist.org quite a bit during your development cycles. A quick win in speeding this up is using a local cache for Packagist which means you’re only downloading across your local network instead of across the Internet.

You can run a local toran proxy instance by running this command:

docker run --name toran-proxy -d -p 82:80 -v /opt/toran-proxy:/data/toran-proxy -e "TORAN_CRON_TIMER=half" -e "TORAN_TOKEN_GITHUB=8ba3e80d61fb" -e "TORAN_PHP_TIMEZONE=Europe/London" -e "TORAN_HOST=toran.local:82" cedvan/toran-proxy:1.1.7-1

We can break this down a little bit. The volume mount means that we have persistence with our cache and settings. We use port 82 and the hostname of toran.local. This means that we can add the following the to our projects composer.json:

"repositories": [
    {"type": "composer", "url": "http://toran.local:82/repo/packagist/"},
    {"packagist": false}
]

Your local composer will now begin to use your toran proxy instance. You can read more information about the Docker image used here over at the Github repo or on the Docker hub page.

You can read the first part of this series detailing an NPM cache here.

Google Doc's Will Support Save as ePub

2016-03-08T18:03:19-00:00

This is a nice feature being added to Google docs that will allow you to save your document as an ePub file. You can read more over on the Google Apps Blog.

PHP UK Postcode Manipulation Library

2016-03-04T18:00:13-00:00

I’ve added a composer installable package for the excellent UK postcode library written by Stephen Morley.

You can find the original package over on his site.

Laravel & Lumen: Return Correct Error Responses for Requests

2016-03-03T18:00:05-00:00

I’ve seen a few pieces of API code dotted around the web recently where people have written simple JSON APIs but their error handling still returns HTML instead of JSON for 404 errors and the likes. This is pretty simple to solve with Laravel or Lumen and easily testable too. The cleanest way I’ve seen of handling this was written by Paul Redmond in his book ‘Writing APIs with Lumen’ which you can purchase on Leanpub.

In your error handler, typically located at app/Exceptions/Handler.php, you’ll need to add a check for wantsJson() on the Request object, like this:

<?php

namespace App\Exceptions;

use Exception;
use Illuminate\Validation\ValidationException;
use Illuminate\Auth\Access\AuthorizationException;
use Illuminate\Database\Eloquent\ModelNotFoundException;
use Symfony\Component\HttpKernel\Exception\HttpException;
use Laravel\Lumen\Exceptions\Handler as ExceptionHandler;
use Symfony\Component\HttpFoundation\Response;

class Handler extends ExceptionHandler
{
    /**
     * A list of the exception types that should not be reported.
     *
     * @var array
     */
    protected $dontReport = [
        AuthorizationException::class,
        HttpException::class,
        ModelNotFoundException::class,
        ValidationException::class,
    ];

    /**
     * Report or log an exception.
     *
     * This is a great spot to send exceptions to Sentry, Bugsnag, etc.
     *
     * @param  \Exception  $e
     * @return void
     */
    public function report(Exception $e)
    {
        parent::report($e);
    }

    /**
     * Render an exception into an HTTP response.
     *
     * @param  \Illuminate\Http\Request  $request
     * @param  \Exception  $e
     * @return \Illuminate\Http\Response
     */
    public function render($request, Exception $e)
    {
        if ($request->wantsJson()) {
            $response = [
                'message' => (string) $e->getMessage(),
                'status' => 400
            ];

            if ($e instanceof HttpException) {
                $response['message'] = Response::$statusTexts[$e->getStatusCode()];
                $response['status'] = $e->getStatusCode();
            } else if ($e instanceof ModelNotFoundException) {
                $response['message'] = Response::$statusTexts[Response::HTTP_NOT_FOUND];
                $response['status'] = Response::HTTP_NOT_FOUND;
            }

            if ($this->isDebugMode()) {
                $response['debug'] = [
                    'exception' => get_class($e),
                    'trace' => $e->getTrace()
                ];
            }

            return response()->json(['error' => $response], $response['status']);
        }

        return parent::render($request, $e);
    }

    /**
     * Determine if the application is in debug mode.
     *
     * @return Boolean
     */
    public function isDebugMode()
    {
        return (boolean) env('APP_DEBUG');
    }
}

Pretty simple and effective, you can handle more cases within your handler as well, but that’s enough to illustrate the idea. You can test the above is working by using something similar to this test:

class ExampleTest extends TestCase
{
    /**
     * A basic test example.
     *
     * @return void
     */
    public function testExample()
    {

        $this->json("get", "/page-that-does-not-exist")
             ->seeJsonStructure([
                 "error"
             ]);

    }
}

A quick and simple tip, but one that seems to be overlooked often.

Making Local Development Faster with Caches for NPM & Composer (Part One: NPM)

2016-03-02T17:00:12-00:00

This is the first post in a two part series detailing how you can use Docker to run local caches of both remote NPM resources and composer packages.

docker pull keyvanfatehi/sinopia

After the image has been pulled down, you can run it and map to a port using the following syntax:

docker run -p 81:4873 keyvanfatehi/sinopia

This will run sinopia and bind the external interface to port 81. There is a fair few configuration options that you can specify, I’d recommend getting a copy of the configuration file then mounting that as a volume.

Next, at the top of the screen, it’ll give you the two commands to make your client version of NPM connect to your new shiny cache, it’ll be something similar to:

If you run those two commands, it’ll set your client to use your new cache and you’re good to go.

The following links are useful reference for this post:

[https://github.com/rlidwka/sinopia

](https://github.com/rlidwka/sinopia) [https://hub.docker.com/r/keyvanfatehi/sinopia/

](https://hub.docker.com/r/keyvanfatehi/sinopia/) [https://github.com/rlidwka/sinopia#using-private-packages

](https://github.com/rlidwka/sinopia#using-private-packages) https://blog.dylants.com/2014/05/10/creating-a-private-npm-registry-with-sinopia/

bcrypt Passwords for WordPress

2016-03-01T18:35:52-00:00

Great development from the Roots team to make WordPress use bcrypt for it’s password hashing. You can read more on their blog post here.

MacOS Sierra Asking for SSH Passphrase

2016-03-01T08:59:47-00:00

There’s a recent change in the handling of SSH keys in MacOS Sierra that means you need to enter your passphrase each time a key is required. There’s an open Radar issue about it and it’s intended behaviour now.

You can work around it by implementing one of the solutions at the links below, I personally went with adding:

ssh-add -A 2>/dev/null;

into my .zshrc file and that solved it for me (or .bash_rc if you use bash).

More information and solutions can be found at:

Anbu: Laravel Profiling Tool

2016-02-27T16:06:41-00:00

If you’re a Laravel user and you’ve been looking for something to profile your application, try Anbu. It’s created by a friend, Dayle Rees and is currently in early beta but would be great for you to try and help out testing.

Laravel Blade Stacks

2016-02-25T18:06:22-00:00

One thing that I often forget about is a nice feature in Laravel that allows you to push scripts or other blocks of code into another part of your templates, so for example:

@push('scripts')
    <script src="/example.js"></script>
@endpush

You can do this multiple times and each one will be pushed on to the stack, when you’re ready to dump and display them, you can dump to view using @stack:

<head>
    <!-- Head Contents -->

    @stack('scripts')
</head>

Nothing ground breaking here, but incredibly useful. You can read more about it on the Laravel documentation.

How do bank payments work?

2016-02-23T13:16:50-00:00

This was a pretty interesting read on how bank payments work.

Phan by Etsy – PHP7 Based Static Analysis

2016-02-20T17:01:57-00:00

Liking Phan by @etsy for static analysis. PHP7 based, works well.

Approaches to Testing

2016-02-20T08:33:18-00:00

I came across this blog post this week which details approaches to testing, check it out:

http://codingitwrong.com/2016/02/08/approaches-to-testing-a-survey.html

Inspecting SQLite Databases in Appcelerator Titanium Projects

2016-01-23T11:35:58-00:00

If you’re using SQLite for your project and you’d like to query the database directly. For Android based devices or emulators, you can do the following:

You’ll need to remember your database name, if you’re using commonJS, you can do something similar to:

exports.getDatabase = function() {
    return Ti.Database.open('app.db');
};

Next, find out what devices you have connected using adb:

adb devices

This will return a list similar to this:

List of devices attached 
10.0.0.2:5555    device

Next, use the ADB shell command to boot into the device:

adb -s 10.0.0.2:5555 shell

This will drop you into a shell on the device. Next, find the database file. In the first code snippet here, you can see our database name is app.db, so we’ll use a combination of find and grep to find the file on the device:

cd /; find * | grep app.db

And this will return the SQLite file and it’s journal, like this:

root@vbox86p:/ # cd /; find * | grep app.db
data/data/com.example.audit/databases/app.db
data/data/com.example.audit/databases/app.db-journal

Now you can use the CLI tool to open the database, like so:

sqlite3 data/data/com.example.audit/databases/app.db

And you’ll be dropped into the CLI tool. You can get a list of tables by using .tables and exit the program via .quit.

The reference for this post came from this wonderful blog post.

Virtualmin Migrated Site Error 'AuthGroupFile'

2016-01-19T19:37:21-00:00

I noticed with the Virtualmin export feature that the Apache htpasswd sometimes doesn’t carry through when you import the package, you’ll end up with an error similar to this:

[Tue Jan 19 10:54:22.551737 2016] [core:alert] [pid 28447] [client 1.2.3.4:55007] /home/website/public_html/.htaccess: Invalid command 'AuthGroupFile', perhaps misspelled or defined by a module not included in the server configuration

To fix this error, you’ll need to enable the authz_groupfile module, for example on Ubuntu, you can use this command:

a2enmod authz_groupfile

Restart Apache and it’ll start working as usual again.

Find Your Most Used Bash Commands

2016-01-17T10:36:24-00:00

If you want to see what your most used commands are within Terminal (will work on Linux too). You can run this command:

history | awk '{print $2}' | sort | uniq -c | sort -rn | head

Fix: Titanium & GenyMotion Player Not Found

2016-01-06T07:33:35-00:00

In the recent update of GenyMotion (Android Emulator) there’s a change which breaks its compatibility with Titanium Studio. You’ll need to open Terminal and create a symlink to fix it, start by opening Terminal and running:

cd /Applications/Genymotion.app/Contents/MacOS/

and then type:

ln -s player.app/Contents/MacOS/player

Shut Titanium Studio and GenyMotion both down and then restart Titanium Studio and it should work as usual after that point.

Real IP for Nginx / Laravel Forge when using Cloudflare

2016-01-04T12:11:43-00:00

If you use Cloudflare, you’ll need to configure Nginx to provide the real IP addresses, you can do this by installing

apt-get install nginx-extras

Then edit /etc/nginx/nginx.conf and within the http block add:

#CloudFlare
set_real_ip_from 199.27.128.0/21;
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 188.114.96.0/20; 
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 104.16.0.0/12;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 2400:cb00::/32;
set_real_ip_from 2606:4700::/32;
set_real_ip_from 2803:f800::/32;
set_real_ip_from 2405:b500::/32;
set_real_ip_from 2405:8100::/32;
real_ip_header CF-Connecting-IP;

Save the file, restart Nginx and then you’ll see the real user IP address within your app. You can read more about this on Cloudflare’s website.

Remember to keep an eye on the Cloudflare IP list for changes.

Email Settings

2016-01-01T10:50:05-00:00

Each email box can store a max of 3GB if you’re using IMAP – but if you add the IMAP account to your phone and computer, they’ll stay in sync (ie, read messages, new mail etc). If you’re using POP3, it downloads the messages permanently to your computer and nowhere else (first to fetch gets the message).

Username:

Email address for the email box you’ve been given

Password:

Password for the email account you’ve been given

Incoming/outgoing servers name:

mail.privateemail.com

Incoming server type:

IMAP or POP3

Incoming server (IMAP):

993 port for SSL, 143 for TLS/STARTTLS

Incoming server (POP3):

995 port for SSL, 110 for TLS/STARTTLS

Outgoing server (SMTP):

465 port for SSL, 25 or 26 for TLS/STARTTLS

Outgoing server authentication should be switched on, SPA (secure password authentication) must be disabled.

You can also access webmail securely at https://privateemail.com.

Git Commit with A Random whatthecommit.com Message

2015-12-30T11:37:45-00:00

Probably shouldn’t advise people to do this:

git commit -m `curl --silent http://whatthecommit.com/index.txt`

That’ll commit your current state with a random message from the fantastic whatthecommit.com website.

Docker + rtorrent = Docktorrent

2015-12-29T12:08:16-00:00

I download a lot of linux images, by far the quickest way is via torrent. Recently discovered a nice implementation of rtorrent in a Docker image, via docktorrent, simple enough to run:

docker run -itd 
    --restart=always 
    --dns 8.8.8.8 
    -p 31405:80 
    -p 45566:45566 
    -p 9527:9527/udp 
    -v /path/to/mount/rtorrent:/rtorrent 
    -e UPLOAD_RATE=1024 
    kfei/docktorrent

The 31405 is the public facing port for the web interface, make sure that you forward both 9527 and 45566 as it’ll need those to run.

Prevent OSX From Opening Photos or Image Capture Automatically

2015-12-24T11:17:50-00:00

Came across this OSX tip recently which will prevent Photos or Image Capture:

defaults -currentHost write com.apple.ImageCapture disableHotPlug -bool YES

Handling HTML Checkboxes with a default value

2015-12-14T08:35:35-00:00

I’ve used this tip myself previously, it’s useful if you want to set a default for a checkbox, such as false:

Add an input of type “hidden”, and set the value to false before you declare the one with your true value, so for example, you could do this:

<input type="hidden" name="agree_to_terms" value="false">
<input type="checkbox" name="agree_to_terms" value="true">

There was this blog post that reminded me of this.

Recursive FTP Get with wget

2015-12-14T07:33:31-00:00

Pretty useful for downloading everything in an FTP account:

 wget -m --tries=0 --continue --server-response --timeout=0 --retry-connrefused ftp://username:password@server.tld

Type Hinting in PHP7

2015-12-08T07:58:27-00:00

I’m happy that PHP7 has finally been released, I’ve been wanting to correctly return type hint for such a long time. You can now do this comfortably:

<?php
declare(strict_types=1);

function getAge(int $age) : int {
    return $age;
}

echo var_dump(getAge(43));
?>

But more usefully, you can type hint interfaces to guarantee the classes you get returned (which is actually far, far more useful and beneficial).

Paypal View IPN Log

2015-11-25T08:32:27-00:00

If you’re a Paypal user and use their IPN functionality, the link to get to the history page is hard to find. You can find it at:

https://www.paypal.com/uk/cgi-bin/webscr?cmd=_display-ipns-history

Google Search Quality Ratings

2015-11-21T16:30:40-00:00

Interesting new document provided by Google which details their search quality ratings. It’s nice to see them actually release something like this officially.

Tip: Add a prefix/suffix to each line of a file

2015-10-29T20:28:24-00:00

Pretty neat tip, I needed to add a prefix of ‘ufw deny from’ to a list of IP addresses in a file, you can do it easily with awk, using this:

awk '$0="ufw deny from "$0' original.list > updated.list

And your updated.list file will now have each line prefixed. You can move the string around to get a suffix too.

Tip: Find String in PHP Files

2015-10-23T13:24:49-00:00

Helpful for finding strings in PHP files:

find . -type f -name '*.php' -print0 | xargs -0 grep 'shell_exec'

Replace a RAID Hard Disk in OSX Based Servers

2015-10-21T18:55:32-00:00

I recently had to replace a hard disk in our OSX server at work. It had been a while since I had done this so I noted down the steps here to run through and change it. Work out which disk in your machine is faulty and replace it with a new drive.

To get started, open up terminal and run:

sh-3.2# diskutil list

This will give you a list of all the disks in your machine:

/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *1.0 TB     disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                  Apple_HFS Macintosh HD            999.9 GB   disk0s2
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *1.0 TB     disk1
   1:                        EFI EFI                     209.7 MB   disk1s1
   2:                  Apple_HFS Server HD               999.3 GB   disk1s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk1s3
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 TB     disk2
   1:                        EFI EFI                     209.7 MB   disk2s1
   2:                  Apple_HFS Untitled 1              2.0 TB     disk2s2
/dev/disk3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 TB     disk3
   1:                        EFI EFI                     209.7 MB   disk3s1
   2:                 Apple_RAID                         2.0 TB     disk3s2
   3:                 Apple_Boot Boot OS X               134.2 MB   disk3s3
/dev/disk4
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 TB     disk4
   1:                        EFI EFI                     209.7 MB   disk4s1
   2:                  Apple_HFS TimeMachine             2.0 TB     disk4s2
/dev/disk5
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS Work                   *2.0 TB     disk5

The new disk in this machine is /dev/disk2. We now need to work out the Unique ID for our RAID array. So we’ll run the following command so that we can get our Unique ID:

sh-3.2# diskutil appleraid list

Which will provide the following output:

AppleRAID sets (1 found)
===============================================================================
Name:                 2TB Mirror
Unique ID:            E07AE421-43C5-4778-B1E3-DC3365F3266B
Type:                 Mirror
Status:               Degraded
Size:                 2.0 TB (2000054943744 Bytes)
Rebuild:              manual
Device Node:          disk5
-------------------------------------------------------------------------------
#  DevNode   UUID                                  Status     Size
-------------------------------------------------------------------------------
-  -none-    82D0A9C3-5ACB-4B3C-A4FC-FD473709E186  Missing/Damaged
1  disk3s2   81D8BD2A-894F-4720-8924-D4654001CF69  Online     2000054943744
===============================================================================

Next, we’ll add this disk to the existing array, the bad disk is listed in the array above as missing, because we’ve replaced it with a new disk.

sh-3.2# diskutil appleraid add member disk2 E07AE421-43C5-4778-B1E3-DC3365F3266B

This will now perform an operation to add this disk to the RAID array:

Started RAID operation on disk5 Work
Unmounting disk
Repartitioning disk2 so it can be in a RAID set
Unmounting disk
Creating the partition map
Adding disk2s2 to the RAID Set
Finished RAID operation on disk5 Work 2012+

And you’re now done for the addition of the drive. You now need to remove the old slice from the array by using this command:

sh-3.2# diskutil appleraid remove 82D0A9C3-5ACB-4B3C-A4FC-FD473709E186 E07AE421-43C5-4778-B1E3-DC3365F3266B

You can now check that your RAID looks good by running:

sh-3.2# diskutil appleraid list

The RAID status should now be showing as online.

Run PHP Script as Daemon Using Supervisord

2015-10-18T11:34:49-00:00

This short post is just a reminder for myself when writing supervisord configuration files, it’s extremely useful for running daemon workers (such as Laravel’s queue workers). You’ll need the supervisor Ubuntu package installed.

Create a file at /etc/supervisor/conf.d/bob-program.conf and enter the file contents:

[program:bob-worker]
command=php /home/bob/worker.php
autostart=true
autorestart=true
user=bob
redirect_stderr=true
stdout_logfile=/home/bob/logs/worker.log

Then restart supervisor with service supervisor restart and your program should launch.

Running Ubuntu Landscape-Server in Docker on Ubuntu 14.04

2015-10-08T20:12:42-00:00

If you use a few Ubunutu servers then you may have come across their management product called Landscape. It’s a pretty complex solution to set up, however with advances in Docker and useful work from the UK Home Office, there is now a Docker image you can use to run it, take a look at this Github repo for more information.

Adding Additional IPs to Linode Servers

2015-10-08T19:55:30-00:00

Update: I no longer use Linode, but I had this draft laying here, so decided to publish anyhow.

If you have an additional IP address on your Linode Ubuntu box, you’ll need to add it to your /etc/network/interfaces file, so open that up using nano or a similar tool, you’ll need to add eth0:1 to the auto line, then an iface block, so that it looks similar to this:

# The loopback network interface
auto lo eth0 eth0:1 eth0:2
iface lo inet loopback

# The primary network interface
iface eth0 inet static
  address 85.159.209.124
  netmask 255.255.255.0
  gateway 85.159.209.1

iface eth0:1 inet static
  address 109.237.26.62
  netmask 255.255.255.0

Save that file, restart networking and then try pinging your new IP address.

Running Ghost in Docker on Ubuntu 14.04

2015-10-08T19:55:07-00:00

If you’re interested in running Ghost for your blog inside Docker. It’s pretty simple to get it up and running.

docker run --name=ghost --restart=always -d -p 123.123.123.123:80:2368 -v /docker/data-volumes/ghost:/var/lib/ghost ghost

I found it better to use a data volume with a mount point of /var/lib/ghost to get it running. The following links were useful in getting up and running:

https://allaboutghost.com/deploying-ghost-with-docker/

https://github.com/docker-library/ghost/issues/7

Running PHPUnit in Isolated Environments with Ubuntu, Docker, Gitlab & Gitlab CI

2015-10-03T17:27:17-00:00

I’m currently working on a new project that houses a framework for multi-tenant applications based on Laravel. The source code is stored in a hosted Gitlab instance via Githost.io – which is run by Gitlab themselves. There’s a set of unit tests that I wanted to run via continuous integration to run the tests with each commit and fast.

Throughout this post, the Docker daemon is running at tcp://192.168.1.254:4243/ instead of via a socket. This is because I use the API for other things over HTTP.

The local development environment is handled by Vagrant, using a custom box based upon Ubuntu that includes Composer, MariaDB and a few other components. This works very well, one simple Vagrantfile can be dropped in to new projects and immediately get running with a LAMP style stack:

Vagrant.configure("2") do |config|
    config.vm.provider :virtualbox do |v|
        v.name = "mult"
        v.customize ["modifyvm", :id, "--memory", 2048]
        config.vm.box = "ssx/lamp"
        config.vm.network :private_network, ip: "192.168.13.37"
        config.ssh.forward_agent = true
    end

    config.vm.synced_folder "./", "/vagrant", type: "nfs"
end

This box works quite well, but to Vagrant up and run tests using this with every commit would be quite slow. Enter Docker. I converted the image that was used in the Vagrant environment to a Docker one and now tests can be ran within seconds instead of minutes.

Gitlab CI has a Docker mode of operation. I couldn’t quite seem to get that up and running well. You can find the instructions for installing a Gitlab CI runner on this page.

I went via the shell route and writing the Docker commands manually to boot up in container and then run our script and tests, in the repo I have build.sh which must be set to +x to execute:

#!/usr/bin/env bash

# Restart the services within the container
service mysql restart
service nginx restart
service php5-fpm restart

# Move into the directory
cd /vagrant

# Run our migrations
php artisan migrate
php artisan db:seed

# Execute our test suite
vendor/bin/phpunit

Next, the .gitlab-ci.yml file needs to be created to run the build and then tests:

stages:
  - test

job1:
    stage: test
    script: 
     - composer install
     - cp .env.example .env
     - php artisan key:generate
     - docker run -d -t -v `pwd`:/vagrant hellossx/lamp:1.0.4 > container.pid
     - docker exec `cat container.pid` /vagrant/build.sh
    tags:
     - docker

This does mean that if tests fail, the container will never get cleaned up. You can add a cronjob to clean up these orphaned containers, with something like this docker-clear-old.sh script:

#!/bin/bash

# Set our docker host
export DOCKER_HOST="tcp://192.168.1.254:4243/"

# Clean up old images that failed tests
for i in $(docker ps -a | grep "hour ago" | cut -f1 -d" "); do docker stop $i && docker rm $i; done

This will remove the containers after an hour and leave the previous ones around for a short while incase you want to attach and investigate anything.

Enabling Docker Remote API on Ubuntu

2015-10-02T22:14:32-00:00

If you need to enable the remote API on an Ubuntu 14.04 based system for Docker, you need to do the following, edit the file located at:

/etc/default/docker

and just under the line that contains something like:

# Use DOCKER_OPTS to modify the daemon startup options.
# DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"

add in your own entry. For example, if you want to listen on 10.0.0.2, port 4243:

DOCKER_OPTS="-H tcp://10.0.0.2:4243"

Then restart docker using:

service docker restart

Confirm it’s now running by pointing your browser at http://10.0.0.2:4243/images/search?term=hello which should return some JSON with image names. You’ll need to set the environment variable for the docker CLI command as well, this should work for you:

export DOCKER_HOST=tcp://10.0.0.2:4243

Then restart your shell and it should work as usual.

Tip: If you’re using New Relic to monitor Docker containers, you’ll need this turned on for it to work properly.

Virtualbox does not start on Mac OSX

2015-09-02T11:27:47-00:00

If you’ve recently upgraded your Mac to OSX El Capitan and you have an issue where Virtualbox has decided to no longer work, try this:

for bin in VirtualBox VirtualBoxVM VBoxNetAdpCtl VBoxNetDHCP VBoxNetNAT VBoxHeadless; do
    sudo chmod u+s "/Applications/VirtualBox.app/Contents/MacOS/${bin}"
done

Found via this link on StackOverflow.

Delete a Remote Git Tag

2015-08-29T12:12:06-00:00

Just a note more to myself as I always forget the syntax of this one. To delete a tag both local and remote:

git tag -d tag_name && git push origin :refs/tags/tag_name

Vagrant + VirtualBox + Update Tools

2015-07-29T20:14:53-00:00

If you’re a Vagrant user that uses Virtualbox to host your machines, sometimes the Virtualbox tools in the guest VM will get out of date. There’s a great Vagrant plugin that will run an update for you. Run the following command to install it:

vagrant plugin install vagrant-vbguest

There is a far more detailed description over on this blog post.

Travel Tip: Send Google Maps Location to Mobile Device

2015-07-21T16:10:57-00:00

If you’re signed in to your Google account, you can click on a pin then click ‘Send to Mobile Device’. This will forward the location over to your device.

Adding Backups to Your Laravel Project

2015-07-07T19:24:21-00:00

If you have a Laravel project and need a simple backup solution, I’d recommend Shawn McCool’s database-backup. Follow the instructions to get that configured to your liking and you can then cron a task like this:

php artisan db:backup --destinationPath=`hostname`-`date +%Y-%m-%d`.sql --destination=dropbox --database=mysql --compression=gzip

To automatically stick a daily backup to Dropbox or any other filesystem provider supported by Flysystem.

Generate Private Keys & CSR for an SSL Cert

2015-07-06T23:05:04-00:00

Something I always tend to have to Google for the right syntax is generating a CSR for an SSL along with a set of keys for it, this post by Google themselves has a good reference:

Generating keys and CSR – Web Fundamentals

Woes of Cloudflare SSL

2015-07-03T18:20:00-00:00

I recently switched tweekly.fm over to serve as SSL via Cloudflare and for the most part, this has been a good transition for most people. There have been the odd reports of people having issues though. I looked into this a bit more and it comes down to the types of certs stored in some phones and browsers – I don’t think they’re out of date per-se, but I don’t think they support a certain cipher, which is what makes them fail.

There’s some good information over on this blog post.

Tweekly.fm Changes

2015-06-23T19:21:12-00:00

If you’re reading this post, you are most likely a user of Tweekly.fm. I’d like to take a little time to write about the changes that have recently happened on the site.

I’d like to note that I have come across some of the most fantastic people in the world through this humble small service and want to see it continue for a long period of time – but to do that things needed to change.

What is Tweekly.fm?

The main functionality of the site takes data that you have provided via Last.fm and posts it, once per week to social networks.

This in itself sounds like a very simple system that has multiple parts within it. Every day, thousands upon thousands of updates are posted to Facebook and Twitter – which take a lot of time and consume a lot of resources to do so. Previously, Google Ads were ran across the site which generated a low amount of income which didn’t cover the cost of keeping the service up and running. The were unattractive and not a good fit for the service.

Crowdfunding

As most of the regular users who use the service know, I attempted to run a crowd funding programme earlier in the year, which did generate enough money to contribute to the hosting costs and some of the development time in updating the site and services that you use.

While the new site has been built, I took time to decide how the new service should run, what the foundations of the service should be and how this can be attained – without financial impact on me personally. I didn’t want to use Google Ads on the new site and as you can see, they’re not present and won’t be.

Support

The biggest cost besides hosting the service is the amount of time answering support tickets and emails. This gets done in my own free time and outside of my 9-5 job. No service is without bugs or hiccups – but a lot of general questions take time to answer and if you’ve ever emailed I’d like to think you got a response quite quickly.

What’s the cost?

The service runs on Linode using a mixture of servers, which for the most part consisted of:

2 x 2GB Linode ($20×2 + $10×2) – Web servers
2 x 2GB Linode ($20×2 + $10×2) – Queue runners, queue daemons
1 x NodeBalancer (1x$20) – Load Balancer
2 x 4GB ($40×2 + 2x$10 Backups) – Database (Master and Slave)

As you see, this comes in at $240/mo which for a service that doesn’t generate a lot of money, is a costly expense that I’ve been happy to cover for a long time. I should also note that the project has very kindly received hosting from Last.fm directly in the past and in the early years this was absolutely key to the project. For the new site, the requirements are about half of those servers.

External Integrations

The last time consumer is keeping up with the external services that are used. Each time Twitter, Facebook and co update their APIs, I needed to change a lot of in-house code to match. During the development of the new system, everything has been moved over to Laravel 5.1 and a lot of standard libraries have been used in the codebase to improve and minimise the amount of time needed for upkeep.

So, what happens now?

The biggest issue for the service was dealing with the amount of traffic that came back to the site once updates are posted, so the new system has no public facing profile pages at the moment. The only place that people can see your updates are on your social media profiles.

Secondly, the largest time consuming task was auto-publishing all the thousands of updates per day – this has now been changed to a premium user feature. You can still sign up for free and continue to post your updates manually – without a charge. This will always remain the case, but for the future auto-posting updates will be limited to those with a subscription.

The changes to the support ticket system are also now in favour of subscribed users. Any support emails from subscribers will be answered first and the rest at some point later in the week.

I know this is quite a change to how the service previously ran and I also accept that some of you may not want to subscribe – I have absolutely no issues with that but I hope you can readdress your expectations in what you want from me.

There has already been a number of folks sign up for premium accounts and for those I’m grateful that you place a value on the service provided.

As always, if you have any questions, ideas or thoughts then please do use the comments below or if you would like to discuss in private, contact support@tweekly.fm and I’ll be happy to talk.

Lastly, thank you to all half a million Tweekly.fm users from around the world – it’s been one hell of a ride so far and I’m glad to have provided you with a service you enjoy.

Migrating Data From Runkeeper to Strava

2015-05-31T13:52:33-00:00

Just a quick note, if you have previous used the Runkeeper platform and want to migrate your data over to Strava, there is a fantastic online tool that will do it for you called tapiriik.com. I’ve used it to bulk import all of my ride data over and it’s been flawless.

Automating Docker on a Gitlab CI Server

2015-05-12T19:44:10-00:00

I’ve currently been setting docker up to run containers for a Gitlab CI server. The basic premise is that the CI server will vagrant up a box, run ansible and finally run phpunit.

This has worked well, however getting the docker containers to destroy afterwards hasn’t been as clean cut.

From using a few other sources, I’m using this script via cron which runs every five minutes to stop and remove any containers that have been running for more than an hour (this is working for our current commit frequency but will need addresses as more things are moved to this CI server).

root@ci-runner:~# cat docker-clear-old.sh 
#!/bin/bash
for i in $(docker ps -a | grep "hour ago" | cut -f1 -d" "); do docker stop $i && docker rm $i; done

This is working quite well so far, I’d be interested in how you’re doing this if you’re a docker user too.

Git: Remove Non Project Files from Repository

2015-04-29T16:36:51-00:00

A quick tip that I always seem to forget, if you need to quickly remove any non project files form a repo, check this Stack Overflow answer out:

http://stackoverflow.com/questions/5037480/removing-non-repository-files-with-git

In short, I think I used this in the end:

git clean -df

Which removed and deleted any files not in the git index.

Ubuntu Server Management with Landscape

2015-04-17T14:56:56-00:00

Both at work and personally, I run a lot of Ubuntu servers. It’s now the standard OS choice for any new servers too. To help with management of these servers, I’ve settled on a combination of New Relic’s server monitoring (and app monitoring for a certain few apps) and the self-hosted/dedicated version of Canonical’s Landscape Dedicated Server.

It’s pretty simple to get set up. You’ll need a server running Ubuntu to install it on. I chose a Digital Ocean $20 droplet.

Create your droplet, run all current updates and check that that hostname is a FQDN. Next, login to the box and switch to root.

add-apt-repository ppa:landscape/15.01
apt-get update
apt-get install landscape-server-quickstart

After this has finished, you will now have an SSL cert at /etc/ssl/certs/landscape_server_ca.crt which you need to copy and make a note of.

root@landscape-server:~# cat /etc/ssl/certs/landscape_server_ca.crt
-----BEGIN CERTIFICATE-----
MIICAzCCAWygAwIBAgIJANbXg3JbmTu9MA0GCSqGSIb3DQEBCwUAMCYxJDAiBgNV
...
[snipped]
...
3fJBHsY6i5i70Gq/5FUV7O7bP7/wnLKIljrO7gI/d5DNEQmXUkgY
-----END CERTIFICATE-----

Make a safe note of the contents of that file. Next, switch over to your client server, and run the following as root:

root@landscape-client:~# mkdir /etc/landscape
root@landscape-client:~# apt-get install landscape-client

This will install the landscape client. Next, create the file /etc/landscape/server.pem and then paste the contents of the server SSL cert into this file.

After this, you need to add the variable ssl_public_key to your Landscape configuration file:

echo ssl_public_key = /etc/landscape/server.pem >> /etc/landscape/client.conf

The final step is to run the configuration utility on the client machine:

root@landscape-client:~# landscape-config --computer-title `hostname` --account-name standalone  --url https://landscape-server/message-system --ping-url http://landscape-server/ping

You’ll be asked a few questions about getting setup and what to allow, tags to use etc. Customise to suit and then register the machine when prompted. If you get a connection error then check you’ve pasted the certificate correctly and added the ssl_public_key variable correctly too.

A couple of links that were useful reference:

http://askubuntu.com/questions/549809/how-do-i-install-landscape-for-personal-use

https://help.landscape.canonical.com/LDS/QuickstartDeployment14.10

PHPStorm: Set your shell to ZSH

2015-03-26T20:39:46-00:00

If you’re a PHPStorm user and also use oh-my-zsh, you can set your default shell to be ZSH in the settings. Head into the preferences, select Terminal, then there is a box for Shell Path, stick in your zsh path – usually /bin/zsh on the mac, like so:

Now when you use the Terminal panel, it’ll be zsh instead of bash:

GPS for Unlocking Train Doors

2015-03-16T18:10:33-00:00

I came across this web page a few days ago which describe an issue with ThamesLink trains having their doors getting stuck due to GPS blackouts. You can read the full article over at ciras.org.uk.

Vagrantfile for Open Source Version of Odoo (formally OpenERP)

2015-03-08T17:37:10-00:00

A couple of colleagues and I have recently been looking at Odoo, there is an open source version which you can self host. There didn’t seem to be any recent Vagrant based versions of this, so I’ve put one together which you can clone from:

git clone https://github.com/ssx/vagrant-odoo

Then run:

vagrant up

It’ll then get the latest install script, run it and get the environment set up. Theres more information in the readme.md file within the repo as well.

Find Installed Packages on Ubuntu (and dpkg) Systems

2015-03-06T22:55:02-00:00

To get a list of all the installed packages on an Ubuntu or dpkg based server, you can use the following command:

dpkg --get-selections | grep -v deinstall

Using an External SMTP Server with Gitlab

2015-01-10T18:57:02-00:00

If you run a Gitlab server and you need to use a custom SMTP service, there is a good set of instructions over at Elijah Paul’s blog:

http://elijahpaul.co.uk/using-an-smtp-server-with-gitlab/

Recursively Update File & Directory Timestamps

2014-12-29T20:15:01-00:00

If you need to update the modified timestamp of a set of files recursively, you can use the following:

find . -exec touch {} ;

Cronjob an RBL Check

2014-12-28T12:39:10-00:00

There’s a fantastic website that you can run curl to fetch a list of the blacklist and simple mail checks to ensure any of your servers that send mail aren’t blacklisted.

Create a new file at /etc/cron.daily/rblchecker and place these contents in:

#!/bin/sh
/usr/bin/curl checkrbl.com

Then make it executable chmod +x /etc/cron.daily/rblchecker and it’ll run. You may need to adjust the path to curl. When it runs, you’ll get the following output:

===================================
CheckRBL.com - Comments, suggestions and bug reports: feedback@checkrbl.com
IP CHECK FOR 85.159.213.127
===================================

RBL Checks
-----------------------------------
 - not listed in Backscatterer
 - not listed in Barracuda Central
 - not listed in McAfee
 - not listed in SenderScore
 - not listed in SpamCop
 - not listed in SpamHaus PBL
 - not listed in SpamHaus SBL
 - not listed in SpamHaus XBL
 - not listed in UCE-Protect

Reverse Record Check
-----------------------------------
Reverse Record present: aeson.ssx.io

SMTP port check
-----------------------------------
Connected to 85.159.213.127 on port 25, mail banner says: aeson.ssx.io.
Found 'A' Record for aeson.ssx.io: 85.159.213.127
DNS 'A' record for aeson.ssx.io matches the given IP 85.159.213.127

Results
-----------------------------------
GOOD: Reverse record matches SMTP banner, 3-way mailcheck PASS.

Gitlab Droplet on Digital Ocean – Add a robots.txt

2014-12-25T17:32:13-00:00

If you’re using Digital Ocean’s Gitlab droplet, it’ll give you a fully configured instance for you to start using straight away. The most often case for people using Gitlab is for private repositories, if that’s the case for you then it’s worth creating a file at:

/opt/gitlab/embedded/service/gitlab-rails/public/robots.txt

and then putting a disallow rule in such as:

User-Agent: *
Disallow: /

Although not a massive security feature it should stop your public facing stuff from showing up on search engines.

State of Tweekly.fm – December 2014

2014-12-13T18:01:38-00:00

It’s been quite a few months for the tweekly.fm project. The original plan was to rewrite the user facing side of the service. Instead of just this, the entire platform was rewritten. The API’s that power the publishing of updates and collection of user data were rewritten from the ground up.

The site and APIs have also been moved over the new servers and architecture and now are loadbalanced which allows around 10,000 connections a second – this will mean you should no longer get timeouts on Sundays and more updates can be sent.

You will now have one tweekly.fm account that you can add connections to. This includes support for Twitter, Facebook, Tumblr, WordPress, AppDotNet and more to come. Each one of these services can be independently configured.

This has enabled me to begin work on developing iOS and Android apps that can interact with the tweekly.fm service. This includes managing your account and posting new updates from your phone.

There will be two versions of the tweekly.fm app. The free version will allow premium users to manage their account and post updates across any of their linked accounts. There will also be a low cost ($0.99/£0.69) version which allows you to create updates on your phone – independent of the tweekly.fm service.

The people who donated earlier this year will receive a free upgrade to a premium account on the new platform and will be able to use the free version of the app. So that you’re aware, enough was raised earlier this year for about two weeks work on the service and platform.

If you’re interested in beta testing the new iOS app then please complete this the form at http://beta.tweekly.fm and I’ll add you to the list.

The launch date for both the new apps and website is going be 11th January 2015. This is due to a few things that still need ironing out with the frontend of the website – although the apps may land in the app store before this date.

I’m still looking for sponsors/advertisers for the site as well, if you’re interested please contact support@tweekly.fm. I’ve also been considering using crowdfunding. If you would be willing to participate in crowdfunding more development of tweekly.fm, then please post in the comments and I’ll see how much reaction there is and whether to set something up.

Fix Messages Stuck Offline on OSX

2014-11-29T20:14:01-00:00

Update: Nino (see comments) has written a blog post on how to automate this a little, read it at http://rubyengineer.com/posts/2

If you have an issue with Messages on OSX being stuck in ‘Offline’ mode that won’t connect, open up Terminal and type the following:

ps auxwww | grep imagent

This will give you output similar to:

scott             341   0.0  0.3  2502924  12828   ??  S     1:41pm   0:04.75 /System/Library/PrivateFrameworks/IMCore.framework/imagent.app/Contents/MacOS/imagent
scott           23666   0.0  0.0  2432772    460 s003  R+    9:25am   0:00.00 grep imagent

The number you’re looking for is 341, which is the process ID of imagent.

Now, type:

kill -9 341

and the process will be restarted, allowing you to change status to online.

SQLSplitter

2014-10-21T17:36:32-00:00

If you need to split an SQL file into smaller chunks, I found this little utility called sqlsplitter that will do exactly that. You can download it from http://rodo.nl/index.php?page=mysql-splitter.

Pretty simple use, stick your SQL file into a directory along with the executable and then run:

sqlsplitter your-sql-filename.sql 5000000

Where 5000000 is the number of bytes you want to split files into.

Using VMware vSphere/ESXi as a Provider for Vagrant

2014-10-11T16:05:48-00:00

Please note: This post assumes you are comfortable with Vagrant, Virtualisation and generic admin tasks on a server.

vSphere & ESXi

One thing I’ve been wanting to do for a while was control my VMWare ESXi based hosts with Vagrant, so that I could easily create and remove boxes while testing. By default, Vagrant can’t do this. Similar to how the interaction with Digital Ocean works, you can install a plugin to aid in getting this to work.

The vagrant-vsphere plugin has been created by the National Snow and Ice Data Center team and open sourced for others to use. You can read more information about it on the Github page for the project at github.com/nsidc/vagrant-vsphere/. There are more configuration options that I don’t mention here that you can tinker with once you are more familiar with how the setup works.

To start, grab the ISO file of your distribution of choice. I chose to go with Ubuntu 14.04 as it is an LTS version and more recently the projects I’ve been working on have used it. You’re going to create a Virtual Machine that will serve as a base for all of your Vagrant instances that you’ll create in future.

Upload the Ubuntu ISO to your datastore and remember where you placed it. Next, create a new virtual machine with a reasonable amount of RAM and CPUs for your box, select the ISO in the CDROM options and boot up your new virtual machine as normal. Set up the box as you usually would do, I left pretty much everything as default apart from the user which you’ll need to setup for Vagrant to use.

During the install you will be prompted to create a user account. Use the username vagrant and the password vagrant, this is required for vagrant to run later down the line.

Complete the installation process, installing any additional software that you would like such as Nginx, Apache, MySQL etc.

After the machine has rebooted, login and run the update commands:

sudo apt-get update -y
sudo apt-get upgrade -y

Install all the updates and then reboot:

sudo shutdown -r now

Next, install the open-vm-tools package which provides VMWare integration for the guest:

sudo apt-get install open-vm-tools -y

Again, after they’re installed, reboot:

sudo shutdown -r now

Next, you need to add the vagrant user to the sudoers file:

sudo su -

Enter the vagrant user password that you set to vagrant earlier and then run the following command to edit the sudoers for the machine:

visudo

At the end of the file, add the line:

vagrant ALL=(ALL) NOPASSWD:ALL

Exit the editor by pressing CTRL+X, Save changes. Next we need to add Vagrants key to the user account:

mkdir -p /home/vagrant/.ssh
wget --no-check-certificate https://raw.github.com/mitchellh/vagrant/master/keys/vagrant.pub -O /home/vagrant/.ssh/authorized_keys

After the file has downloaded, reset permissions to the correct ones:

chmod 0700 /home/vagrant/.ssh
chmod 0600 /home/vagrant/.ssh/authorized_keys
chown -R vagrant /home/vagrant/.ssh

Now we’ll need to install the OpenSSH server (if you haven’t already):

sudo apt-get install -y openssh-server

After installation is complete, edit the configuration file at /etc/ssh/sshd_config and make sure that following options match the below:

Port 22
PubKeyAuthentication yes
AuthorizedKeysFile %h/.ssh/authorized_keys
PermitEmptyPasswords no

Restart the SSH server:

sudo service ssh restart

Now we have our template box. In your vSphere client, right click on your newly created virtual machine, select Template, then select “Clone to Template”. Follow the steps and give it a good name that you’ll remember.

As you can see in the image above, I called mine ubuntu.template.dc.sw10.net which is sitting in the resource pool Linux in the host 192.168.1.2 and that host resides in the datacentre dc.sw10.net. This is just a sample configuration and you can adjust yours to suit.

Installing the Vagrant Plugin

If you’ve got this far, I’m going to assume that you have Vagrant installed on your on your machine. Run the following command to install the plugin that we’re going to use:

vagrant plugin install vagrant-vsphere

You can learn more about the plugin and it’s various options over on the Github page at github.com/nsidc/vagrant-vsphere/. There are quite a few settings that you can change which I won’t detail here.

Creating a Vagrantfile

To create a Vagrantfile that we can run, we’ll need to add a few configuration options in to instruct the plugin on where to locate things. I started with the following Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = 'vsphere'
  config.vm.box_url = 'https://vagrantcloud.com/ssx/boxes/vsphere-dummy/versions/1/providers/vsphere.box'

  config.vm.provider :vsphere do |vsphere|
    # The host we're going to connect to
    vsphere.host = '192.168.1.253'                            

   # The host for the new VM
    vsphere.compute_resource_name = '192.168.1.2'            

    # The resource pool for the new VM
    vsphere.resource_pool_name = 'Linux'                    

    # The template we're going to clone        
    vsphere.template_name = 'ubuntu.template.dc.sw10.net'    

    # The name of the new machine
    vsphere.name = 'bingo'                                     

    # vSphere login
    vsphere.user = 'root'                                    

    # vSphere password
    vsphere.password = 'password'                            

    # If you don't have SSL configured correctly, set this to 'true'
    vsphere.insecure = true                                    
  end
end

Save that file, double check your credentials and then run Vagrant:

 vagrant up --provider=vsphere

You can see progress in the task/event log in vSphere, but you should see the template being cloned and then the machine booting:

After this has completed, you’ll have something similar to the following output where you ran vagrant up:

 ➜  vagrant up --provider=vsphere     
 Bringing machine 'default' up with 'vsphere' provider...
 ==> default: Calling vSphere CloneVM with the following settings:
 ==> default:  -- Template VM: dc.sw10.net/vm/ubuntu.template.dc.sw10.net
 ==> default:  -- Target VM: dc.sw10.net/vm/bingo
 ==> default: Waiting for SSH to become available...
 ==> default: New virtual machine successfully cloned and started
 ==> default: Rsyncing folder: /Users/scott/Downloads/tmp/ => /vagrant

You can now use the other Vagrant commands such as ssh:

➜  tmp  vagrant ssh     
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-32-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

 System information disabled due to load higher than 1.0

Last login: Sat Oct 11 13:30:21 2014 from 192.168.1.3
vagrant@ubuntu:~$ hostname
ubuntu
vagrant@ubuntu:~$

and of course you can vagrant destroy too:

 ➜  vagrant destroy
==> default: Calling vSphere PowerOff    
==> default: Calling vSphere Destroy

I’ll most likely update this article soon with provisioning Nginx, PHP etc. If you need a dummy box to use with vagrant-vsphere, then I’ve added one to Vagrant Cloud at:

https://vagrantcloud.com/ssx/boxes/vsphere-dummy/versions/1/providers/vsphere.box

OSX: Format an Encrypted Drive

2014-09-23T20:21:20-00:00

If you need to format an encrypted disk on OSX, you’ll need to use the command line diskutil as Disk Utility won’t let you do anything with the device.

First, find the ID of the disk you wish to format, using:

diskutil list

Which will list your disks, for example I wish to format the 2TB drive, I’d use disk2

/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *250.1 GB   disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                  Apple_HFS Macintosh HD            249.2 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *1.0 TB     disk1
   1:                        EFI EFI                     209.7 MB   disk1s1
   2:                  Apple_HFS Data                    477.0 GB   disk1s2
   3:                  Apple_HFS Scott                   522.8 GB   disk1s3
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 TB     disk2
   1:                        EFI EFI                     209.7 MB   disk2s1
   2:                  Apple_HFS Time Machine            2.0 TB     disk2s2

We can then go ahead and format the drive (remember this will format your drive and remove all data from the disk):

diskutil eraseDisk JHFS+ Test disk2

Replace the label Test with the name you wish to give the new partition. After you’ve done this, you can edit the disk normally with Disk Utility.

Install Mailcatcher on Laravel Homestead

2014-09-20T19:34:09-00:00

Update – October 18th 2014:

The wonderful Chris Fidao noted in the comments that if you’re not using Laravel Homestead and are using a different box, you may need to install the build-essential package too. You can do that with:

sudo apt-get install build-essential

After that’s installed the instructions below should be fine.

If you’re looking to debug outbound email easily, then mailcatcher will be a life saver for you. It’ll give you a nice interface to view outbound emails:

To install it, ssh to your box and follow these steps:

sudo apt-get install ruby-dev libsqlite3-dev

Once that’s installed, you can install the gem with this command:

sudo gem install mailcatcher

After mailcatcher is installed, you can run the program with:

mailcatcher --http-ip 0.0.0.0

If you’ve using Vagrant then you’ll need to forward port 1080 from the guest to the host machine as well.

Add Logfile to Memcache/Memcached Daemon on Linux

2014-09-11T22:34:55-00:00

If you need to add a log file for memcache to help debug an issue, you can either use the following syntax:

memcached -d -m 3072 -l localhost -p 11211 -u nobody -v 2>>/tmp/memcached.log

or if you’re on RHEL/CentOS, you can edit the file /etc/sysconfig/memcached like so:

PORT="11211"
USER="memcached"
MAXCONN="3048"
CACHESIZE="256"
OPTIONS="-vv >> /var/log/memcached 2>&1"

Save the file, restart memcache and you’ll have a log file. This was taken from this StackOverflow question.

Windows: Add user to group via Command Line

2014-09-10T17:47:19-00:00

I recently had to fix a Windows machine that had a broken user profile and needed to add the new user to the admin group.

A few Google searches later and this is the command to run in the command line prompt:

net localgroup group user /add

You can find this and more syntaxes at the helpful windows-commandline.com.

Slack Beta for Mac with Multiple Team Support

2014-09-07T21:44:48-00:00

If you’re a Slack user (and you should be) and wish to use the Mac app, it’s a bit tedious for those of us who are in multiple teams. They’ve recently released a beta version that has support for multiple teams in a much better interface.

You can download it from HockeyApp at this URL:

https://rink.hockeyapp.net/apps/06bd6493684f65a3b8f47aca92c9006e

Syncable Host File to Make the Internet Suck Less

2014-08-20T21:05:16-00:00

We’ve all been there. A friend sends you a link to an internet shock site. If you’re familiar with the hosts file on your machine, then you can use this large sync-able list to compliment your own rules. If you using a Mac, you can use the nifty Gas Mask program to edit your hosts file without having to jump into a shell.

URL for the hosts file is:

http://someonewhocares.org/hosts/hosts

As well as IPv6 files, there are a few other formats available as well.

Virtualmin – FTP/SFTP File Writing Errors

2014-08-19T20:58:36-00:00

If you’re using Virtualmin on your servers and have a few odd FTP error messages (happened for me using SFTP too), then it’s one of three things.

Disk quota for the hosting account has exceeded
Server has ran out of memory
The configuration options below need to be set in /etc/proftpd.conf

As mentioned above, I found that these two options added into /etc/proftpd.conf fixed the file write issue in some cases:

AllowRetrieveRestart on
AllowStoreRestart on

Useful Notes for phpinfo() & Capturing Output of phpinfo()

2014-08-16T20:44:45-00:00

I recently needed to capture the output of phpinfo() for a to a log upon install. I’d been able to do this in the past but couldn’t quite remember how I managed it. This is where ob_start() and ob_flush() come into their own.

To get the contents of phpinfo(); into a variable, you can use the following:

ob_start();
phpinfo();
$strPhpInfo = ob_get_contents();
ob_clean();

The variable $strPhpInfo will now contain the output from phpinfo().

A side note, if you only want the variables section of phpinfo(), then use:

phpinfo(32);

and it’ll give you just that in the output. You can read more about the other values you can pass to phpinfo() at php.net/manual/en/function.phpinfo.php.

PHP ActiveRecord – Get Last Query

2014-08-15T21:17:14-00:00

I recently had to do some work on an older site that used the PHP implementation of ActiveRecord. One of the issues I had was that the queries being ran weren’t produced the expected result. I found two commands to help with this an issue the last run query:

If you’re within a model, you can do:

echo self::connection()->last_query;

and if you’re outside your model, you can use:

echo model::connection()->last_query;

Install Old Version of Laravel 4.1.X for PHP 5.3 Servers using Composer

2014-08-10T21:23:38-00:00

If you need to install an older version of Laravel for use with a server that has PHP 5.3.X for example, you can do so using the following command:

composer create-project laravel/laravel project-directory 4.1.*

This is useful when you can’t upgrade the version of PHP but still wish to use Laravel.

Almost Time for Amsterdam!

2014-08-09T14:16:30-00:00

It’s a couple of weeks away until I head off for Amsterdam.

I’ll arrive in Amsterdam on the 25th August of August and depart on the
31st. I’ll be in Amsterdam to attend the fantastic
Laracon EU conference along with a work colleague.

We’re currently in the process of making plans of what to do while
we’re there, we got a lot of the common tourist things done last year, so
this year it’ll be good to see some new places.

If you have experience of renting a boat of in the 'dam too, I’d like a chat.

If you’ve got any suggestions, leave them in the comments below. If you
know anyone in Amsterdam with good local knowledge, point them in the direction
of this post too!

Interactivity for Payment Forms

2014-07-26T13:41:13-00:00

Interactivity for Payment Forms

A short while ago now I read an article by Michael Villar of Stripe on using animations when users are completing payment forms.

Villar put forward the argument that this improves the user experience.

Below is a demonstration of doing that.

Laravel: Filter Every Array Element with a Closure

2014-07-24T21:37:56-00:00

I always forget about how awesome Laravel Collection’s are. I came across this gem when needing to run a filter on every element of an array. I think it may of been Dayle’s book that first informed me of them. For example, if you have an array of fruits that you want to filter down, you can do the following:

// Create an array of fruits
$array = ["banana", "apple", "orange"];

// Transform the array into a Collection object
$collection = new IlluminateSupportCollection($array);

// We don't like banana's anymore, so we're going to filter them out
$no_bananas = $collection->filter(function($element) {
    if ($element != "banana") return true;
});

// Dump out our array now, and we'll see the banana's are gone
dd($no_bananas);

Laracon US 2014

2014-07-17T22:04:31-00:00

If you haven’t noticed yet, the Laracon US 2014 videos have been posted online for a short while now, you can grab them over on Userscape’s website at http://userscape.com/laracon/2014/.

One talk of note, which I’ve recommended to a lot of people is by Greg Baugues, titled “Devs and Depression”.

Adding Google Sitemap's to Ghost Blogging Platform

2014-07-16T19:06:57-00:00

I recently wanted to replace my statically generated sitemap.xml file with a dynamically one generated by ghost. It ended up being pretty trivial to do, but the following two links helped immensely. At present, it’s a hack to the core which you’ll have to repeat if you upgrade – but that’s a small price to pay for having your sitemap generated dynamically.

For adding sitemap support:

http://tstrimple.com/hacking-ghost-adding-dynamic-sitemap-xml/

For updating the format/count:

http://www.mathachew.com/blog/customizing-your-dynamic-sitemap-in-ghost/

MariaDB: Set Maximum Server Connections via Query

2014-06-28T20:17:30-00:00

If you’re a MariaDB user and need to set the max_connections variable a little higher while the server is running, you can use the following SQL query:

# This will give you the current mac_connections setting
select @@max_connections;

# Set the maximum connections allow for MariaDB
set global max_connections = 200;

# To see that the variable has been changed, run the following query again:  
select @@max_connections;

ncftp: Recursive File Get

2014-06-18T21:29:55-00:00

A while ago I wrote a post about using the basic ftp command to recursively fetch remote FTP files, it seems this doesn’t completely fetch them recursively. A great little program called NCFTP will do this accurate and pretty quickly. Install NCFTP if you haven’t already, then run the program:

ncftp

This will give you a prompt, at which type the following:

set save-passwords yes

Which will allow the program to save your password while its working and transferring files. Open a connection to your FTP server by using the following at the command prompt:

open -u ftp_username ftp_host

It’ll ask for a password, enter it and then it’ll go away and connect. To then download your files recursively, enter the command:

bgget -r file/path

and once you’ve finished queuing files, type:

bgstart

to begin the transfer and NCFTP will go ahead and download all your queued files.

Titanium: Remove All Children of View

2014-06-06T21:04:27-00:00

I picked up this function somewhere a while ago, it’s useful when you want to remove all the children elements of a view in Titanium.

function removeAllChildren(obj) {
    var c = obj.children.slice(0);
    for (var i = 0; i < c.length; ++i) {
        obj.remove(c[i]);
    }
}

Apple Swift Programming Language Manual Download

2014-06-02T20:47:41-00:00

After Apple’s impressive WWDC keynote earlier, you can download the reference manual for the Swift programming language for iBooks using the following URL:

https://itunes.apple.com/gb/book/swift-programming-language/id881256329?mt=11

Linux: Find All Modified Files in Last X Days

2014-05-30T14:24:33-00:00

Nifty linux command to remember:

$ find . -mtime -3 -ls /

This will display all the files modified in the last 3 days. You can change 3 to suit as many days as you require.

How to Detect Version of CS Cart Websites

2014-05-27T21:59:00-00:00

Quick tip, if you need to find the version of a CS Cart based website, simply append ?version to the query string, so for example if you shop name is mysupershop.co.uk, point your browser at:

http://www.mysupershop.co.uk/index.php?version

You’ll be presented with a page that notifies you which version of CS Cart is installed and running, similar to:

CS-CART: version 2.1.4 PROFESSIONAL

Laravel London Meetup

2014-05-24T12:12:32-00:00

On the 15th May, a work colleague and I attended Laravel London for a couple of talks by Aussie Kirk Bushell and Maks Surguy. Both were great presentations, Kirk spoke about abstracting validation away to use across your application and Maks spoke about the pitfalls of ajax file uploads.

If you’re interested in attending one of the meetups you can find the group on Meetup.com at London Laravel which is created by the fantastic Jonty Behr.

Laravel Homestead: Add Enviroment Variables to Nginx

2014-05-24T01:08:50-00:00

If you wish to set your environment by using server environmental variables with the new Laravel Homestead you can do the following:

Edit the website configuration name, by running the following command within the VM:

$ sudo nano /etc/nginx/sites-available/local.dev

Within the location ~ .php$ { block, update your configuration to resemble this:

location ~ .php$ {
    fastcgi_split_path_info ^(.+.php)(/.+)$;

    fastcgi_param LARAVEL_ENVIRONMENT "dev";

    fastcgi_pass unix:/var/run/php5-fpm.sock;
    fastcgi_index index.php;
    include fastcgi_params;
}

Now restart Nginx

$ sudo /etc/init.d/nginx restart

Next, edit your bootstrap/start.php file and update the method to detect environment:

$env = $app->detectEnvironment(function(){
    return getenv("LARAVEL_ENVIRONMENT");
});

Now you can set your environments per server.

MySQL Procedure: Drop All Tables in Database

2014-05-24T00:12:41-00:00

If you ever need a progmatic way to drop all tables in a database, you can use the following stored procedure:

DELIMITER $$

DROP PROCEDURE IF EXISTS `drop_all_tables_from` $$

CREATE PROCEDURE `drop_all_tables_from`(IN schema_target VARCHAR(128))
BEGIN
    DECLARE table_list TEXT;

    SELECT
        GROUP_CONCAT(`TABLE_NAME`)
    INTO
        table_list

    FROM `information_schema`.`TABLES`
    WHERE
          `TABLE_SCHEMA` = schema_target;

    IF table_list IS NOT NULL THEN
        SET @drop_tables = CONCAT("DROP TABLE ", table_list);

        PREPARE stmt FROM @drop_tables;
        EXECUTE stmt;
        DEALLOCATE PREPARE stmt;
    END IF;

END $$

DELIMITER ;

To use, simply call the procedure with the database name as the parameter:

CALL drop_all_tables_from("database_name");

Note: This will drop all tables, without confirmation for the database name provided.

Laravel: Prevent Sessions for Routes via a Filter

2014-05-22T18:41:22-00:00

I needed to prevent Laravel from using sessions for a few API endpoints in an app, this is how I did it:

Route::filter('session.remove', function()
{
    return Config::set('session.driver', 'array');
});

This can be used quite easily with a route or route group:

Route::group(array(
    "prefix" => "api/v1",
    "before" => array("session.remove")
), function() {
    Route::get('/search', 'SearchController@getPerformSearch');
});

Useful Slides from Laracon NYC 2014

2014-05-18T17:38:31-00:00

I’d picked up a couple of the slides published for this years Laracon event in New York.

Amazon Web Services for Artisans by Jeremy Lindblom

Async PHP with React by Jeremy Mikola

Company Unable to Fulful Your Order? Aquire It.

2014-05-18T11:58:59-00:00

I absolutely love data and robots.

I came across a story this morning while reading Reddit. An Amazon warehouse staff member had taken part in AMA sessions which you can read here.

One on the questions posed was about how much automation is present in the warehouse. The person replied with a detailed answer which included the fact that Amazon placed an order for a Kiva automation system that Kiva was unable to fulfil.

Instead of going to find alternatives, Amazon purchased Kiva and now gets their orders fulfilled first and Kiva’s other customers afterwards.

These Kiva robots are absolutely awesome and it turns the pickers work into a doddle.

Laravel: Inverse Database Seeder

2014-05-17T22:19:23-00:00

This is a pretty useful package to create database seed files from an existing database.

Laravel Inverse Seed Generator:

https://github.com/orangehill/iseed

FOI: Vacant Commercial Properties in Newcastle under Lyme

2014-05-11T10:31:36-00:00

A recent Freedom of Information request that I had placed at the local council has been returned. There were a few notable buildings currently vacant which I will document further soon.

Below is a map detailing the currently vacant commercial properties that are owned by Newcastle under Lyme Borough Council.

Map was produced with the useful jHERE mapping library that I’ve been wanting to use for a while now. You can find the code used to produce the map on Github at github.com/ssx/data-foi-nulbc-4332 where you can fork/use it as required.

Mac OSX/iOS Tip: Add Folder of eBooks to iOS Device

2014-05-05T10:49:34-00:00

You can add a whole folder of eBooks to your iOS device by opening iTunes, select the files and then drop them onto the device name. It’ll take a few seconds to copy over (or minutes if you have a lot of books).

Stuffing JavaScript into DNS TXT Records

2014-04-23T22:40:36-00:00

This is from the slightly interesting notebook.

If you check out the TXT record for dor.ky you’ll see there is a record

stuffed with Javascript, which will fire in a lot of whois web services.

scott$   dig txt dor.ky @sam.ns.cloudflare.com

;; QUESTION SECTION:
;dor.ky.                IN  TXT

;; ANSWER SECTION:
dor.ky.            300 IN  TXT "<script type='text/javascript'>alert('This is from a DNS record!');</script>"

You can see this firing in action over at http://mxtoolbox.com/SuperTool.aspx?action=txt%3Ador.ky, I’m not picking on them specifically, there were a number of online tools that were vulnerable to this.

I’d be interested to see if you have any more creative ideas on what we can do with this.

Malware Detection for Linux Servers using Maldet

2014-04-21T13:33:30-00:00

Maldet is a malware scanner that’s useful on servers that accept file uploads. I use it especially on servers where I have WordPress sites hosted and it alerts me to the fact that people have used themes with Malware tucked away inside.

You can visit the Maldet website for a more detailed description.

Installation will only take a few moments and it’s a handy tool to have. It’ll also install a cronjob that will run automatically each day and alert you of any potential or found issues.

Fetch the current version of Maldet

[root@bob]#  cd /tmp
[root@bob]#  wget http://www.rfxn.com/downloads/maldetect-current.tar.gz

Unzip

[root@bob]# tar -zxvf maldetect-current.tar.gz

Installation

[root@bob]# cd maldetect-*
[root@bob]# ./install.sh

Edit Configuration File

[root@bob]# nano /usr/local/maldetect/conf.maldet

Run an on-demand Maldet scan

[root@bob] maldet /home

Lessons Learned When Building a SaaS

2014-04-19T14:37:58-00:00

A recent post by the saasclub.com newsletter had a good article in it by Clément Vouillon which was titled “9 Lessons Learned Building SaaS”. Effectively they had asked a few members of eFounders what the lessons they learned when building their services.

You can read the full article at efounders.co/9-lessons-learned-building-saas/.

What You Should Do About the OpenSSL/Heartbleed Security Problem

2014-04-09T18:09:03-00:00

By now you have probably read the news and seen the warnings about the OpenSSL Heartbleed security vulnerability that is present in certain versions of the software that powers 66% of the internet.

Here’s what you should now do:

If you’re not a server admin then check the websites you frequently use at http://filippo.io/Heartbleed/. If the site shows as ‘seems safe’ then login and update your passwords to something secure and that you have not used before. Nearly every large tech company has recommended resetting your passwords.

Read the BBC article Heartbleed Bug: Public urged to reset all passwords for more information.

As noted in the above BBC article:

The University of Surrey’s Prof Alan Woodward is among security experts to have suggested internet users should now update their login details.

He suggests the following rules should be observed when picking a new password.

Don’t choose one obviously associated with you

Hackers can find out a lot about you from social media so if they are targeting you specifically and you choose, say, your pet’s name you’re in trouble.

Choose words that don’t appear in a dictionary

Hackers can pre-calculate the encrypted forms of whole dictionaries and easily reverse engineer your password.

Use a mixture of unusual characters

You can use a word or phrase that you can easily remember but where characters are substituted, eg, Myd0gha2B1g3ars!

Have different passwords for different sites and systems

If hackers compromise one system you do not want them having the key to unlock all your other accounts.

Keep them safely

With multiple passwords it is tempting to write them down and carry them around with you. Better to use some form of secure password vault on your phone.

If you are a server administrator, make sure your openssl libraries are up to date using yum/rpm/apt or your package manager of choice and then test your sites using the tool above.

Tip: Apache mod_rewrite Rule Tester

2014-04-04T22:19:14-00:00

If you’re a #devop then at some point you’ve probably had to deal with Apache’s mod_rewrite. You can find a really handy tester written by Martin Melin on his website.

With his mod_rewrite tester you can test your rules before putting them into staging or production.

Export All Your Flickr Photos with Flump

2014-04-03T22:28:40-00:00

This year I’ve decided to not renew my Flickr Pro membership and then had to the fun task of exporting all my images.

I came across this nifty Adobe Air application called Flump that will run on my Mac.

You can download Flump and export your own images.

Project: Parking in Southampton

2014-04-01T18:38:00-00:00

Content coming soon.

Project: Local Electronic Data

2014-04-01T18:37:22-00:00

Content coming soon.

Project: OAuth Examples

2014-04-01T18:36:45-00:00

Content coming soon.

Project: StokeTraffic

2014-04-01T18:35:26-00:00

Content coming soon.

AirMail for Mac is Awesome – Almost.

2014-03-30T15:14:44-00:00

For a few months now I’ve been using Airmail for Mac as my main email client on my home Mac and Macbook Air. The interface is fantastic and it’s lightweight too which is exactly what I wanted. The whole experience of using the app is comfortable.

The problem comes however with no support for PGP/SMIME and no preference syncing across multiple computers.

Sadly, this means I can’t ditch Apple Mail just yet, but if there are enough people who want these two features within Airmail, I think the developers would implement them.

You can cast your vote for the features over on the Airmail help forum:

Airmail Support for PGP

iCloud Preference Syncing

Image credit: airmailapp.com

Laracon EU 2014

2014-03-28T23:34:59-00:00

Tech companies, this is your chance to get a little free advertising. I’ll be attending the wonderful Laracon EU 2014 conference in Amsterdam this year and I’ll happily wear a t-shirt from your company while I’m there.

Also, if you want to provide some stickers, coupons or other promotional material for me to pass on while I’m there – I’d be happy to.

Pop me an email to scott@dor.ky and I can let you know shipping details etc.

Using Iron.io Workers for Large Scale PHP Posting with Laravel

2014-03-28T22:35:59-00:00

The most popular and prolific codebase I’ve created and worked with has been the best way to post last.fm to twitter, the wonderful tweekly.fm. Each day, the service publishes hundreds and thousands of social media updates to Facebook and Twitter.

The Challenges & Self-DDOSing

The biggest challenge with having a system that posts so many updates is that every time we post an update, there is a link to the users profile which is then visited and spidered. The effect this has on the website and service is massive.

It’s the equivalent of a lower scale Distributed Denial-of-Service attack.

Last.fm Help

In late 2013, tweekly.fm had become massively popular. We had users from some of the largest tech companies in the world using our service. Scalability once again became an issue. Towards the end of the year, Last.fm provided us with a beefier hosting solution which eased the load we would be under daily and allowed the service to expand.

Sequential Posting

To alleviate the DDOS effect the first few versions of the service would post social updates sequentially with a predefined sleep between each outbound post. This worked well when there was only a few hundred users onboard but as the service grew it became impossible to post all social updates within the 24 hour period we required them to go out at.

Multi-curl

One of the first libraries that I’d used to post Twitter updates was twitter-async by Jaisen Mathai. This introduced me to multi-curl and then I discovered rolling-curl. Being able to post multiple tweets was a great advancement, but this brought with it an unintended consequence of amplifying the DDOS feeling for our servers.

In January 2013, we moved away from our Last.fm hosted solution to a dedicated server and a dedicated database server. This allowed massive expansion for a short period of time. We quickly ended up encountering more issues with scalability and more importantly cost. Nearly 99% of our user base consists of free users that are shown advertising. We were fast heading past revenue for server costs.

Resolutions

Around October 2013 I discovered iron.io almost by accident. I’d recently begun rewriting tweekly.fm into the excellent Laravel framework. I was testing Laravel 4’s queuing systems and noticed a reference to iron.io. After reading more into the IronMQ product – I came across IronWorker.

The difference IronWorker provided for tweekly.fm cannot be understated. It allows us the create updates, package them up to be sent and then queue en-masse into an IronWorker queue. These are then processed in batches and an entire days updates can be sent out in a matter of minutes.

Sunday is the busiest day of the week for tweekly.fm. Regularly for a year now, we’ve been pushing out over 200,000 updates. That’s 8,333 updates an hour or 138 a minute. This would take over 24 hours sequentially, around 18 hours with multiple curl calls and takes just over 40 minutes with IronWorker at a fraction of the cost.

I was able to remove one of the servers and save on the hosting cost – this alone reduced our costs by half.

The exceptional service, support and price is worth it alone. Mix that in with the fact costs were halved – I’m not too sure how you can look anywhere else when needing to run PHP workers for your large scale projects.

Markdown Editors for Mac OSX

2014-03-28T20:57:15-00:00

I’ve recently began to use Markdown for almost all of my text writing needs. This includes technical notes, readme documents and Github commits/issues/comments. Even my blog is now powered by Markdown as I’m using Ghost.

One thing I’m missing though is a Mac based Markdown editor. I have iA Writer which I’m still happy with and that also syncs via iCloud. I’d be happier with a program that has an inbuilt preview so that I don’t have to keep opening Marked to see if I’ve got the formatting/flow of the document correct.

I’ve just come across the highly rated Markdown Pro and I’m thinking of giving it a try. Have you used it? Would you recommend it?

If you have any suggestions , please let me know in the comments and I’ll check them out. I don’t mind paying for good, worthwhile software – I’d just like something that can fit my needs of writing and a ‘live’ preview.

Microsoft Word, Excel, Powerpoint and One Note for iPad (and free!)

2014-03-27T20:38:57-00:00

If you’ve missed the news today, Microsoft has finally released an iPad version of Word, Excel, Powerpoint and One Note on the app store.

“Microsoft is focused on delivering the cloud for everyone, on every device. It’s a unique approach that centers on people — enabling the devices you love, work with the services you love, and in a way that works for IT and developers,”

Satya Nadella, Microsoft

I was quite surprised to see that they’re free too.

You can download them via the app store.

Google Apps Toolbox

2014-03-25T11:03:54-00:00

A neat feature I came across this week was the Google Apps toolbox. If you’re a user of Google Apps then this can be very useful to track down issues and configuration problems.

The toolbox currently features a browser debugger, DNS verification and couple of log analysers.

You can find the toolbox at https://toolbox.googleapps.com/apps/main/.

Fix Artisan Command Errors with 'Allowed memory size .. bytes exhausted'

2014-03-20T19:48:18-00:00

If you encounter an error resembling the following:

Fatal error: Allowed memory size of 117647092 bytes exhausted (tried to allocate 14 bytes) in /home/v_102/Laravel/database/connection.php on line 192

PHP has run out of memory while performing Eloquent operations. The biggest cause of this is the query logging that Laravel does behind the scenes. You can either increase memory_limit to a higher amount, however a better solution is to add this line before you start your database queries:

DB::disableQueryLog();

Preparing for Laracon EU 2014

2014-03-20T08:04:20-00:00

It’s the time of year again when I start to plan the summer months and the conferences I’ll be attending. Last year I attended the wonderful Laracon EU at Bim Huis and it was a magnificent venue and the content of the conference was great.

I’m currently in the process of planning this years trip, which I’ll be taking a few extra days before the conference to do tourist type things in Amsterdam. If you have any suggestions for places and things to do for tech minded people, let me know in the comments.

How to Exclude IP Address(es) from Google Analytics

2014-01-27T11:59:53-00:00

If you’re like most developers and you need to exclude yourself from showing in a Google Analytics profile, then you can set your IP address in the exclude list so that you don’t skew your analytics.

To do so, login to Google Analytics, select the account your wish to edit and then click the Admin tab.

Login to Analytics
Select Admin
Select the Account that you wish to edit
Click ‘All Filters’
Click ‘New Filter’
Enter name ‘Filter my IP’
Select Exclude, then ‘Traffic from the IP Addresses’, then ‘that are equal to’ and enter your IP address in the boxes provided.
Select the views you wish to apply this to and then click ‘Add’.
Once finished, click ‘Save’.

View Column Numbers in Excel instead of R1:C1 Letters

2014-01-21T10:55:24-00:00

Ever wanted Excel to display column numbers instead of letters (think CSV imports!):

Microsoft Excel can be configured to display column labels as numbers instead of letters. This feature is called “R1C1 Reference Style”, and though it can be useful, it can also be confusing if inadvertently enabled.

You can select between the two modes quite easily. Read more at https://kb.wisc.edu/page.php?id=781.

Heart Internet: Switch PHP Version to 5.4 & 5.5

2014-01-09T15:54:09-00:00

If you’re a Heart Internet customer and have a project that needs to run a newer version of PHP, you can add a SetEnv flag to your .htaccess file to switch the version on the fly.

For PHP 5.5 use:

SetEnv DEFAULT_PHP_VERSION 55

For PHP 5.4 use:

SetEnv DEFAULT_PHP_VERSION 54

You can also switch back a version down to 5.2 and 5.1.

Dump or Display SSL Certificate Information on Mac OSX/*nix

2013-11-16T22:02:09-00:00

The command below will display information about an SSL certificate to STDOUT.

openssl x509 -in certificatefile.crt -noout -text

List of Amazon AWS IP Addresses for Firewall Configurations

2013-11-16T21:46:33-00:00

A recent project used the fantastic iron.io platform to process background jobs (highly recommended if you need to write anything at scale). Part of the task required access back across the Internet to a server behind a firewall.

The iron.io platform runs on top of Amazon AWS so finding the IP addresses took a little bit of searching, but then I came across a forum post with a good list.

US East (Northern Virginia)
72.44.32.0/19 (72.44.32.0 – 72.44.63.255)
67.202.0.0/18 (67.202.0.0 – 67.202.63.255)
75.101.128.0/17 (75.101.128.0 – 75.101.255.255)
174.129.0.0/16 (174.129.0.0 – 174.129.255.255)
204.236.192.0/18 (204.236.192.0 – 204.236.255.255)
184.73.0.0/16 (184.73.0.0 – 184.73.255.255)
184.72.128.0/17 (184.72.128.0 – 184.72.255.255)
184.72.64.0/18 (184.72.64.0 – 184.72.127.255)
50.16.0.0/15 (50.16.0.0 – 50.17.255.255)
50.19.0.0/16 (50.19.0.0 – 50.19.255.255)
107.20.0.0/14 (107.20.0.0 – 107.23.255.255)
23.20.0.0/14 (23.20.0.0 – 23.23.255.255)
54.242.0.0/15 (54.242.0.0 – 54.243.255.255)
54.234.0.0/15 (54.234.0.0 – 54.235.255.255)
54.236.0.0/15 (54.236.0.0 – 54.237.255.255)
54.224.0.0/15 (54.224.0.0 – 54.225.255.255)
54.226.0.0/15 (54.226.0.0 – 54.227.255.255)
54.208.0.0/15 (54.208.0.0 – 54.209.255.255)
54.210.0.0/15 (54.210.0.0 – 54.211.255.255)
54.221.0.0/16 (54.221.0.0 – 54.221.255.255)
54.204.0.0/15 (54.204.0.0 – 54.205.255.255)
54.196.0.0/15 (54.196.0.0 – 54.197.255.255)
54.198.0.0/16 (54.198.0.0 – 54.198.255.255)

US West (Oregon)
50.112.0.0/16 (50.112.0.0 – 50.112.255.255)
54.245.0.0/16 (54.245.0.0 – 54.245.255.255)
54.244.0.0/16 (54.244.0.0 – 54.244.255.255)
54.214.0.0/16 (54.214.0.0 – 54.214.255.255)
54.212.0.0/15 (54.212.0.0 – 54.213.255.255)
54.218.0.0/16 (54.218.0.0 – 54.218.255.255)
54.200.0.0/15 (54.200.0.0 – 54.201.255.255)
54.202.0.0/15 (54.202.0.0 – 54.203.255.255)

US West (Northern California)
204.236.128.0/18 (204.236.128.0 – 204.236.191.255)
184.72.0.0/18 (184.72.0.0 – 184.72.63.255)
50.18.0.0/16 (50.18.0.0 – 50.18.255.255)
184.169.128.0/17 (184.169.128.0 – 184.169.255.255)
54.241.0.0/16 (54.241.0.0 – 54.241.255.255)
54.215.0.0/16 (54.215.0.0 – 54.215.255.255)
54.219.0.0/16 (54.219.0.0 – 54.219.255.255)

EU (Ireland)
79.125.0.0/17 (79.125.0.0 – 79.125.127.255)
46.51.128.0/18 (46.51.128.0 – 46.51.191.255)
46.51.192.0/20 (46.51.192.0 – 46.51.207.255)
46.137.0.0/17 (46.137.0.0 – 46.137.127.255)
46.137.128.0/18 (46.137.128.0 – 46.137.191.255)
176.34.128.0/17 (176.34.128.0 – 176.34.255.255)
176.34.64.0/18 (176.34.64.0 – 176.34.127.255)
54.247.0.0/16 (54.247.0.0 – 54.247.255.255)
54.246.0.0/16 (54.246.0.0 – 54.246.255.255)
54.228.0.0/16 (54.228.0.0 – 54.228.255.255)
54.216.0.0/15 (54.216.0.0 – 54.217.255.255)
54.229.0.0/16 (54.229.0.0 – 54.229.255.255)
54.220.0.0/16 (54.220.0.0 – 54.220.255.255)
54.194.0.0/15 (54.194.0.0 – 54.195.255.255)

Asia Pacific (Singapore)
175.41.128.0/18 (175.41.128.0 – 175.41.191.255)
122.248.192.0/18 (122.248.192.0 – 122.248.255.255)
46.137.192.0/18 (46.137.192.0 – 46.137.255.255)
46.51.216.0/21 (46.51.216.0 – 46.51.223.255)
54.251.0.0/16 (54.251.0.0 – 54.251.255.255)
54.254.0.0/16 (54.254.0.0 – 54.254.255.255)
54.255.0.0/16 (54.255.0.0 – 54.255.255.255)

Asia Pacific (Sydney)
54.252.0.0/16 (54.252.0.0 – 54.252.255.255)
54.253.0.0/16 (54.253.0.0 – 54.253.255.255)
54.206.0.0/16 (54.206.0.0 – 54.206.255.255)

Asia Pacific (Tokyo)
175.41.192.0/18 (175.41.192.0 – 175.41.255.255)
46.51.224.0/19 (46.51.224.0 – 46.51.255.255)
176.32.64.0/19 (176.32.64.0 – 176.32.95.255)
103.4.8.0/21 (103.4.8.0 – 103.4.15.255)
176.34.0.0/18 (176.34.0.0 – 176.34.63.255)
54.248.0.0/15 (54.248.0.0 – 54.249.255.255)
54.250.0.0/16 (54.250.0.0 – 54.250.255.255)
54.238.0.0/16 (54.238.0.0 – 54.238.255.255)
54.199.0.0/16 (54.199.0.0 – 54.199.255.255)

South America (Sao Paulo)
177.71.128.0/17 (177.71.128.0 – 177.71.255.255)
54.232.0.0/16 (54.232.0.0 – 54.232.255.255)
54.233.0.0/18 (54.233.0.0 – 54.233.63.255)
54.207.0.0/16 (54.207.0.0 – 54.207.255.255)

GovCloud
96.127.0.0/18 (96.127.0.0 – 96.127.63.255)

Varnish Crashing Regularly on Cent OS 6.1, 6.2, 6.3 and 6.4

2013-11-11T18:47:28-00:00

Fix:

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

jQuery Validate Method: notEqual

2013-11-07T20:20:58-00:00

I often have the need with jQuery validate to test whether or not a value is equal to a predetermined one. The method below provides that functionality:

jQuery.validator.addMethod("notEqualTo", function(v, e, p) {  
  return this.optional(e) || v != p;
}, "Please specify a different value");

Which can then be used in a similar fashion to this:

$("form").validate({  
  rules: {
    nameField: { notEqualTo: "Placeholder Text Name" }
  }
});

Tip - Laravel 4, 500 Error on Install

2013-10-29T00:19:46-00:00

If you install Laravel 4 and you’re using Apache with a certain Virtual Host configuration (specifically using VirtualDocumentRoot in my case) then you’ll need to include a RewriteBase directive within the .htaccess file in public/ to correct it.

An example of a fixed .htaccess file:

<IfModule mod_rewrite.c>  
    Options -MultiViews
    RewriteEngine On
    RewriteBase /
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule ^ index.php [L]
</IfModule>

Laravel 4 Queue Names

2013-10-26T21:49:27-00:00

Just a quick note, it looks as though you can’t use a forward slash in the name of a beanstalkd queue name.

Makes Laravel behaviour a little weird and random errors.

Internet Explorer, Ajax Requests and UTF-8

2013-10-20T15:37:27-00:00

While debugging a legacy application recently, I came across an ajax script that was returning the header below and failing in IE:

Content-Type: application/json; charset=utf8

The correct header, has a dash between the UTF and 8 (correctly so):

Content-Type: application/json; charset=utf-8

Hopefully this can help someone out if they have to debug something similar.

Linux/Mac Recursive FTP Using Built-in FTP Programme

2013-10-05T08:01:41-00:00

I had to fetch a few thousand files across a few directories on a remote server recently and I was going to use NCFTP, however I’d remembered that the stock ftp program on linux can in fact do a recursive fetch. You need to remember to turn the interactive prompts off as well, because if you don’t then you will have to press the ‘Y’ key to download each file.

# Open connection  
ftp user@host.com

# Change directory
cd to/the/path

# Turn the interactive prompt off
prompt off

# Use the command 'mget' to fetch the files, the glob (*) will execute
# on the remove server
mget *

# Exit cleanly
exit

MySQL Views are Extremely Useful

2013-10-04T22:45:42-00:00

A quick overview of using views in MySQL.

First, create our products table:

CREATE TABLE `products` (  
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` text COLLATE utf8_unicode_ci,
  `price` decimal(10,2) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

Now create our users table:

CREATE TABLE `users` (  
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(45) COLLATE utf8_unicode_ci DEFAULT NULL,
  `email` varchar(45) COLLATE utf8_unicode_ci DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

Lastly, create our transaction table:

CREATE TABLE `transactions` (  
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `user_id` int(11) DEFAULT NULL,
  `product_id` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `product_id_idx` (`product_id`),
  KEY `user_id_idx` (`user_id`),
  CONSTRAINT `product_id` FOREIGN KEY (`product_id`) REFERENCES `products` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
  CONSTRAINT `user_id` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

To query across these tables we would need to join them. A query such as:

SELECT  
    users.id AS UserId,
    users.name AS UserName,
    users.email AS UserEmail,
    products.id AS ProductId, 
    products.name AS ProductName, 
    products.price AS ProductPrice 

FROM transactions 

LEFT JOIN users ON (users.id = transactions.user_id) 

LEFT JOIN ON (products.id = transactions.product_id);

If we need to run this query often, we could simplify the process by creating a view of the above query. To create the view, we run the following SQL:

USE `my_database`;

CREATE VIEW `view_transaction` AS  
    SELECT 
        users.id AS UserId, 
        users.name AS UserName, 
        users.email AS UserEmail, 
        products.id AS ProductId, 
        products.name AS ProductName, 
        products.price AS ProductPrice 

    FROM transactions 

    LEFT JOIN users ON (users.id = transactions.user_id) 

    LEFT JOIN products ON (products.id = transactions.product_id);

Instead of then having to query against the large join, we could simply query the view:

SELECT * FROM view_transaction WHERE ProductPrice > 10

PHP Interfaces Quickstart

2013-09-29T18:47:50-00:00

PHP interfaces are still quite new to most. Below is a simple demonstration of their usefulness. I think I’d seen this within one of the many fantastic laravel books.

<?php  
interface BillerInterface {  
    public function bill(array $user, $amount);
}
interface BillingNotifierInterface {  
    public function notify(array $user, $amount);
}


class NotifyByEmail implements BillingNotifierInterface {  
    public function notify(array $user, $amount)
    {
        echo "Send email for ".$user["name"]." of $amount".PHP_EOL;
    }
}


class NotifyByMessage implements BillingNotifierInterface {  
    public function notify(array $user, $amount)
    {
        echo "Display message for ".$user["name"]." of $amount".PHP_EOL;
    }
}


class NotifyBySMS implements BillingNotifierInterface {  
    public function notify(array $user, $amount)
    {
        echo "Send SMS for ".$user["name"]." of $amount".PHP_EOL;
    }
}


class StripeBiller implements BillerInterface {  
    public function __construct(BillingNotifierInterface $notifier)
    {
        $this->notifier = $notifier;
    }

    public function bill(array $user, $amount)
    {
        // Bill the user via Stripe...
        echo "Bill via Stripe".PHP_EOL;
        $this->notifier->notify($user, $amount);
    }
}


class PaypalBiller implements BillerInterface {  
    public function __construct(BillingNotifierInterface $notifier)
    {
        $this->notifier = $notifier;
    }

    public function bill(array $user, $amount)
    {
        echo "Bill via Paypal".PHP_EOL;
        $this->notifier->notify($user, $amount);
    }
}


class EPDQBiller implements BillerInterface {  
    public function __construct(BillingNotifierInterface $notifier)
    {
        $this->notifier = $notifier;
    }


    public function bill(array $user, $amount)
    {
        echo "Bill via ePDQ".PHP_EOL;
        $this->notifier->notify($user, $amount);
    }
}


$newBill = new StripeBiller(new NotifyBySMS);
$newBill->bill(
    array(
        "name" => "Scott Wilcox"
    ),
    "123.89"
);
$newBill = new PaypalBiller(new NotifyByMessage);
$newBill->bill(
    array(
        "name" => "Scott Wilcox"
    ),
    "123.89"
);
$newBill = new EPDQBiller(new NotifyByEmail);
$newBill->bill(
    array(
        "name" => "Scott Wilcox"
    ),
    "123.89"
);

Handy Tip: Use an Apple plug to charge a Lumix battery

2013-09-28T09:42:00-00:00

Came across this tip by a work colleague the other day, if you’re a
traveller and use Apple devices, you can use the Apple multi plug instead of
having to carry a figure-8 cable around with you as well.

Solr, Tomcat and UTF-8

2013-09-23T08:52:33-00:00

I had to fix an issue recently where Apache Solr wasn’t returning any results for German words. After altering the schema to accommodate the German language, the same issue of being unable to search for German words was still there. It turns out that earlier version of Apache Tomcat aren’t UTF-8 enabled by default, it’s a configuration option that you need to explicitly set the URI encoding used.

<Connector port=“8080” protocol=“HTTP/1.1” connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8" />

You can read more on the Tomcat Wiki.

PHP: How to List All Functions of an Extension

2013-05-04T19:00:35-00:00

Recently I’ve been doing quite a lot of work that required UUID
generation. There is two PHP libraries for generating UUID’s. One of them
hasn’t been updated in a long, long time and the second one has absolute
no documentation for it. After spending a few moments trying to guess the
function names that this newly compiled extension provided, I found that PHP has
a command called get_extension_funcs() which does exactly what it says on the
tin. It will provide you with a list of functions that an extension can provide.

By running:

<?php  
echo var_dump(get_extension_funcs("uuid"))  
?>

We get the following output:

Tech at Christmas

2013-01-14T14:01:22-00:00

Over the holiday period this year, I decided to get a few things off my to-do list that have been sitting around for a while. I decided the contents of my Christmas geekery would be worth of a blog post and that is the one you’re reading now.

Replicating Replication

After having a number of database servers dotted around the world, with different backup systems and schedules – I finally got around to implementing replication for two of them. This will allow me to immediately roll over to the backup slaves if the main servers ever go down. The beauty of the system is that I can also run some queries on the slaves instead of the masters, which means less sluggyness for the master when I want to generate certain reports. Performance has improved for both the master (less large queries being run against it) and provides a piece of redundancy that I didn’t have previously. I’m considering doing the same for the rest of the MySQL servers I administer too. Duplicating backups of data can never be a bad thing and if one of the backups can be used for a warm start then all the better.

Centralised Syslog

Part of the reason for the large reports is that I’ve recently configured all syslog daemons to log to a centralised server. This means I can track errors more easily and also run custom queries against the syslog database with SQL instead of needing to trawl the log files. I’ve been highly impressed with the system so far and it certainly has made a big difference configuring the replication so that I can run reports against the slaves and see processes that are causing a lot of errors.

Hardware

I picked up a HTC Wildfire recently too with the aim of unlocking, rooting and replacing the bootrom within the device. I managed to succeed in doing all three of them and it wasn’t too difficult a process. The end result was an updated OS on the slightly older HTC Wildfire than was possible with their stock ROMs. The hardware for the Wildfire isn’t the best spec but the phone works fine and the OS is coming along well. I don’t think I’m interested in making the jump to Android but I’m very tempted by the Nokia Lumia 920. That’s another debate to have. The only other piece of hardware that I had flashed over Christmas was an old Fonera router, which now has dd-wrt and runs the wifi upstairs at home.

The only other hardware related things I got done over Christmas was to replace the screen and digitiser on an iPhone 4S and the HTC Wildfire. The Wildfire screen replacement was easy, one of the easiest I’ve done. Ordering the correct part is that issue with HTC hardware for me as a large number of their phones share the same components and manufacturing identifiers. The iPhone 4S flex and power button replacement was also easy to do but as with all Apple hardware takes an age to do.

PHP Projects

There was a couple of pieces of PHP I cobbled together over the holidays too. One of the ideas for CKEditor that I had was to show a page highlight when you hover over a link, I used the Grabz.it API to create this jQuery Link Preview. The end result isn’t quite right yet, but its not too far off what I’m aiming for. The second project was a system for importing CSV files (although the end product could be used for anything) into a database. I wanted to provide a visual way of linking the data in the CSV against the fields in the database. Enter the domline plugin and a few pieces of knowledge from around the web (Stackoverflow user Bruno Alexaandre> provided most of the legwork). The end result will be a jQuery & PHP CSV Importer. Ideas and thoughts both welcome on these projects.

PHP: Get Accurate User IP Address

2012-12-08T12:55:30-00:00

// PHP: Get Accurate User IP Address  
function strFindCorrectIP() {  
    $headers = array(
        'HTTP_CLIENT_IP',
        'HTTP_X_FORWARDED_FOR',
        'HTTP_X_FORWARDED',
        'HTTP_X_CLUSTER_CLIENT_IP',
        'HTTP_FORWARDED_FOR',
        'HTTP_FORWARDED',
        'REMOTE_ADDR'
    );

    foreach ($headers as $key) {
        if (array_key_exists($key, $_SERVER) <mark>= true) {
            foreach (explode(',', $_SERVER[$key]) as $ip) {
                if (filter_var($ip, FILTER_VALIDATE_IP) !</mark> false) {
                    return $ip;
                }
            }
        }
    }
}

Taking Advice

2012-12-04T21:04:04-00:00

Over the years I’ve written numerous blog posts both on my own site and around the web too. They have ranged from topics close to home and choices in life that I’ve made, to being focused on programming and technology. I’ve often used my blog to share code snippets and to ask opinion on programming methods. As a person, I’m always keen to listen to others, both in technology and life.

It’s always fascinating to learn how different people approach problems and whether I can learn anything from their methodologies (the answer to that is yes, I’ve learned a lot from others including correcting some of my own practices). I have always believed that you should take on board advice from others, especially those who have worked in the field for as long as you. There are far too many people who are ignorant to the methods of others, which cannot only cause stagnation in projects but also severe repercussions.

Consider the following scenario:

You have a project that has a login system or some form that someone else has written. After investigating further, you find a bug in this system that allows anyone to bypass the login mechanism. You report this to the developer of the application who then is adamant that they have correctly built the mechanism to their previous standards.

The above scenario is entirely plausible, XSS, CORS and other hijacks are evolving all the time, a piece of code that you wrote two years ago could easily be insecure now. In our field and world of technology, we can never sit still. Things evolve constantly on the web and if you don’t change with the times then you’re going to have issues. I’ve spoken to a few people in the past who had the same attitude as mentioned above, their code was ‘secure’ when they wrote it so why should they have to change it now? It’s an all too common attitude within the programming world. A world where the majority of developers feel the need to 1-up each other and prove that they’re the best of the best. I can remember reading a blog post on a similar topic recently, ‘Developer Envy’. This attitude cannot only lead to problems when working in teams but also can be the downfall of major projects if it isn’t managed properly. Every one wants to succeed, that’s part of human nature.

As a developer you have the right to question others and the way in which they have approached problems. When we were kids, we were always the ones who took their toys apart to see how the worked. This undying curiosity is what powers my life and my personality, I always want to know why. The same comes with the projects that I work on, if you have done something better then I’m happy to listen.

Google APIs Error & Access Not Configured

2012-08-11T19:59:22-00:00

If you’re working with the Google APIs, one of the error messages you may come across as:

403: Access Not Configured

All this simply means is that you haven’t set the service to ‘on’ in the APIs console and accepted the terms and conditions. Once you’ve done that then you request will function as expected.

How to Enable Emoji on iOS Devices

2012-07-11T22:50:54-00:00

Step 1.

Step 2.

Step 3.

Step 4.

Step 5.

Step 6.

Step 7.

Step 8.

The Future of Debugging

2012-07-11T22:40:59-00:00

Source: @Grumpydev

3D Views of the London Underground Stations

2012-06-26T21:19:09-00:00

It’s not very often that I come across a site which I find truly amazing. I came across Station Maps by Andrew Godwin this evening and found them completely fascinating. I have a long standing interest in the London Underground system and to see the stations in this way rekindled that greatly.

It’s truly amazing how complex Bank and King’s Cross St. Pancras are.

Oyster Card Travel History Now Online

2012-06-07T02:07:50-00:00

I’ve just had the following email from TFL explaining that they’ve now placed all my Oyster travel journeys into the online portal. Can’t wait to see if they expose this data via the API too.

I am writing to let you know we now offer an enhanced service for checking your journey history online.

You will need an Oyster online account to access the service. If you do not have an account, you can create one by visiting tfl.gov.uk/oyster.

You will be able to see all your journeys made from the date you open your account, whether you use pay as you go, season ticket, Travelcard or a Bus & Tram Pass.

PHP is Mostly CRUD

2012-05-18T21:58:32-00:00

I recently got the book “MongoDB and PHP” by Steve Francia which had an interesting paragraph in the first chapter that talks about how the use of stored data has changed in recent years:

With all the problems with ORMs, you may wonder why programmers use them at all. People were willing to make the compromises to adopt ORMs for one big reason; PHP applications are by and large CRUD applications. Rarely do they use all of the rich features the relational database provides, so giving them up seemed a small price to pay for the benefit of simplified access to the data. Additionally, there weren’t really any other good options. For very simple projects, one could write SQL in one’s code, but this was hard to debug and even harder to ensure that it was done securely. PHP is famous for enabling SQL injection attacks, as inexperienced developers pass variables right into the SQL without sanitization.

Which lead me to think about how my own use of data has changed over the years.

Virtualmin, ProFTPd & Passive FTP

2012-05-13T11:26:09-00:00

The current version of Virtualmin ships with an issue for ProFTPd on CentOS due to two modules that it needs to track connections not being loaded. There is a quick fix for this, you’ll need to edit the iptables-config file located at:

[root@pluto ~]# nano /etc/sysconfig/iptables-config

Add the end of the configuration file, add the following line:

IPTABLES_MODULES="ip_conntrack_netbios_ns ip_conntrack_ftp"

You can now restart the iptables service with the following command and FTP should work through passive mode:

[root@pluto ~]# service iptables restart

Forcing SSL Protocol Using PHP or Apache's mod_rewrite

2012-05-11T20:30:35-00:00

There are quite a few circumstances where you may need to force a site, app or page to serve over a secure SSL connection. In PHP development there are two ways to approach this problem depending on your setup. If you’re using Apache to serve your site and you have the ability to use .htaccess files then you can use a rewrite rule to simplify this task server side.

Using Apache mod_rewrite to force SSL

If you can use the mod_rewrite function below to enforce this server side.

Use PHP to force page to serve over SSL

You may be in the position where you only need to redirect a certain page (although you could easily modify the rewrite rule above to do that) or you don’t have the ability to use .htaccess files then you will need to handle this within your actual PHP script.

Toronto Food & Place Reviews

2012-05-09T17:17:05-00:00

As you probably know, I was in Toronto for a couple of weeks earlier this year. In a new respectful habit I’m trying to force myself into I’m making my views on locations visited know via Foursquare and Google Places. The list below contains all the name worthy places I’d visited and a short review of each venue.

Toronto Pearson International Airport

Having been to Pearson a few times, I can safely say that it’s becoming a fairly known path for me. There is a large number of places to eat, drink and shop throughout the concourses. All of the airport staff are helpful and willing to assist you with anything you need. There are plenty of facilities around and always within reach of the gates. Enjoy every experience I’ve had with this airport and I’m sure most others do too.

Crowne Plaza Fallsview

While visiting Niagara Falls I stayed at the Crowne Plaza Fallsview hotel. The hotel isn’t similar to others in the area and it retains the old aura of a time gone by. The entrance hall, hallways and fittings all reflect the level of service that is provided by the staff. I spent two nights in Niagara Falls and both were pleasant. The bedrooms were clean, tidy and stocked full of helpful items. The service was great and there is a Starbucks at street level. The location is great to wander around Niagara Falls too, with the Maid of the Mist tours just a few minutes walk across the road.

Niagara Falls

For a very long time, Horseshoe Falls has been on the list of places I’ve wanted to visit in the world. While in Canada this time around I had the chance to actually go and stay in Niagara Falls and see the majestic Falls in all their glory. Along with doing the usual tourist things like Maid of the Mist (see below) it was also good to visit Clifton Hill which is very similar to Blackpool and the old English seafront arcades. The entire tourist area is a delight to walk around in with plenty of places to eat, buy and wander around. I was so awestruck by the sheer size and power of the Falls that I would recommend that every person in the world should go see.

Air Canada Centre

I was fortunate enough to see the Red Hot Chilli Peppers play live at the Air Canada Centre whilst I was in Toronto. The venue is in a great location downtown and easy enough to get to by subway, streetcar and by walking. The ACC staff are extremely friendly and I had the pleasant experience of having ‘Lou’ as our usher for the night. Before the event started Lou spoke to myself and others seated nearby about his experience working at the ACC, he had missed two events in 30 years of working there which is an incredible!

The facilities were great, clean and easy to locate. The merchandise for the concert was easy to find too!

Maid of the Mist

Along with seeing Niagara Falls, one of the other things I wanted to do was to take a trip on the world famous ‘Maid of the Mist’. Be careful if you want to go and visit this fantastic attraction and head over to the dock early in the morning. The queues build up extremely quickly and you can end up queuing for hours. The whole process was a pain free experience though, although it’s very wet when you’re in front of both the American and Horseshoe falls.

The trip itself takes around 15 minutes after leaving the dock. If you can get onto the boat quickly then do so, as you’ll want to find yourself a good vantage point. Staff were most friendly and even offered to take photos for me too!

Crescent Beach

Review content.

Old Fort Erie

Review content.

Courtyard by Marriott

Review content.

Coach House Restaurant

Review content.

Teds Collision

Review content.

Steam Whistle Brewing

Review content.

CN Tower

Review content.

Elephant & Castle

Review content.

The County General

Review content.

A&W

Review content.

Frans

Review content.

Pizza Pizza

Review content.

Teriyaki Express

Review content.

Urban Eatery

Review content.

Starbucks

Review content.

BMO Field

Review content.

Tim Hortons

Review content.

Sears

Review content.

Yonge & Dundas Square

Review content.

AMC Cinema

Review content.

The Pickle Barrel

Review content.

Kuala Lumpur Makes Internet Access a Requirement for Food and Drink Outlets

2012-01-21T23:14:16-00:00

A lot of things get my attention in the news now-a-days but a piece from Kuala Lumpur impressed me. A new law has been passed in Kuala Lumpur that stipulates food and drink outlets must now provide free wireless internet in order to be granted or to renew a license to operate. I think this is a fantastic step from the local government as Internet access has been a must have for most people in recent years.

Here in the UK, we’re often lucky that most popular coffee houses such as Starbucks or Costa Coffee and food outlets such as McDonalds have long provided free WiFi. Starbucks do it in a very interesting way and effectively allow members of their reward scheme to have free Internet access whereas non-members have to use BT OpenZone. Registration is free, so there is no reason for you not to sign up and its a valuable marketing tool for Starbucks.

Get Week Start and End Timestamps in PHP

2012-01-15T20:15:58-00:00

For a recent piece of PHP I was writing I needed to find the start and end timestamps for a given week. I came up with the following piece of code to achieve getting the timestamps.

Facebook Feature Request - Event Search to Honour 'center' and 'distance' Parameters

2012-01-07T17:59:03-00:00

For quite a while now one feature I’ve felt was missing from the Facebook Graph API was searching for events by latitude and longitude along with a distance. You can currently issue a search for place objects, but the place object is the only one which honours the center and distance parameters.

I’ve filed a bug report for the feature request for Facebook to implement this and we’ll see how that progresses. If you would like to add your notes to the feature request or show your support you can do so at https://developers.facebook.com/bugs/288653361182652.

Create a New Google Account Without Gmail

2012-01-05T20:06:30-00:00

A couple of times recently I’ve needed to create new Google accounts. If you’ve recently needed to do this you probably noticed that Google’s default form now wants you to effectively create a ‘screen name’ which will come with GMail.

I’d imagine this is a ploy to push more users to Google+ and its a smart way to do it, however if you want to use your existing email address then you will need to use the old formatted signup page which you can find at https://accounts.google.com/SignUpWithoutGmail.

What is a Wharf?

2012-01-01T15:57:36-00:00

When I went to visit Last.fm in
London last year I took some time to wander around the City as I do most times
I’m in London. This particular time I decided to take one of the
riverboats down to Greenwich – for a long time now, Greenwich has been
one of my favourite places in and around London. While I was on the riverboat
the tour guide was discussing local points of interest which they have done many
times before. However, this time he pointed out a sign which said Wharf above a
new apartment block. After then asking all of the people on the riverboat if they
knew the meaning of the word wharf non did. Up to that point in my life even I
hadn’t questioned what the meaning of it was.

As it was explained to us, a wharf is a Ware House At River Front. During the
late 1800s this were prevalent all along the Thames and most other waterside
locations.

Creating and Requesting SSL Certificates

2011-12-30T21:48:14-00:00

One of the less common server tasks during the year is the setup and maintenance of SSL certificates for web services. There has always been the air of dread when this comes around but I’m not too sure where the reason for that comes from. The process is simple enough to go through and you have two choices in how you use SSL.

You can either generate your own certificate which will cause the users browser to prompt (this is by design, you’re not a certificate authority) or you can purchase a certificate from an authority such as Verisign. Most of the larger certificate authorities are accepted by 99% of browsers. There are various levels of certificate validation which include domain validation (providing you own the domain), extended validation (proving you are a business/individual) and enterprise level validation.

The first thing you will need to do is generate your own key if you haven’t done so already. Using the openssl program you can generate your own key using the following command:

openssl genrsa -des3 -out my.key 2048 
This will generate a private key for you using 2,048 bits of encryption. Some certificate authorities require this level of encryption.

Now that you have your own identity key you can create a Certificate Signing Request (CSR). This is a request to an authority to create a certificate on your behalf. The program will ask you a series of questions that you need to answer. The most important of these is the FQDN field which must match the site you’re securing. Again, using OpenSSL you can use this command to generate your CSR:

openssl req -new -key my.key -out my.csr 
If you want to generate your own self signed certificate then you can use the following OpenSSL command to do so. This generates a certificate that is valid for 365 days from today:

openssl x509 -req -days 365 -in my.csr -signkey my.key -out my.crt 
If for any reason you would like to remove the passphrase/password from your private key, you can do so using the command:

openssl rsa -in www.key -out new.key

Enable Keyboard Repeat in OSX Lion

2011-12-24T21:46:43-00:00

An odd bug in OSX Lion means that the keyboard repeat rate options in the Keyboard preference pane doesn’t get honoured. To enable this, you can run the following command at a Terminal prompt:

defaults write -g ApplePressAndHoldEnabled -bool false

Then either logout or restart your machine to force a re-read of the defaults configuration.

Set Your Facebook Username

2011-12-18T22:11:40-00:00

One of the most often asked questions for me when it comes to Facebook is how administrators of pages and profiles can set usernames. In the past, Facebook haven’t really pushed this feature but in the age where social marketing and name based branding is becoming more important – setting your username is a bonus point when it comes to promoting your brand.

You can set your Facebook username at the following URL:

<a href="http://www.facebook.com/username/" target="_new">http://www.facebook.com/username/</a>

Changing Running Styles to Help Your Knee

2011-12-18T22:08:43-00:00

http://www.youtube.com/watch?v=7jrnj-7YKZE
http://www.youtube.com/watch?v=XrOgDCZ4GUo&feature=related

Enable Airdrop for All Macs

2011-11-13T17:01:58-00:00

Fire up a terminal and enter the following to enable Airdrop on all macs:

defaults write com.apple.NetworkBrowser BrowseAllInterfaces 1

Using the Skype Protocol Handler

2011-10-19T19:23:52-00:00

A lot of people use Skype now-a-days and one convenient feature is the ability to use the skype:// protocol handler for creating actionable links on the web. Examples of this are to initiate a call, or add a user as a friend. The list below details the types of links that you can use:

<td>
  Example Link
</td>

<td>
  Result
</td>

<td>
  <a href="dordotky?call">skype:dordotky?call</a>
</td>

<td>
  Skype will initiate a call to the user
</td>

<td>
  <a href="dordotky?add">skype:dordotky?add</a>
</td>

<td>
  Skype will pop up a box to add the user
</td>

<td>
  <a href="dordotky?chat">skype:dordotky?chat</a>
</td>

<td>
  Skype will initiate a chat with the user
</td>

<td>
  <a href="dordotky?userinfo">skype:dordotky?userinfo</a>
</td>

<td>
  Skype will open the &#8216;View Profile' window for the given user
</td>

<td>
  <a href="dordotky?voicemail">skype:dordotky?voicemail</a>
</td>

<td>
  Skype will record a voicemail to leave for user
</td>

<td>
  <a href="dordotky?sendfile">skype:dordotky?sendfile</a>
</td>

<td>
  Skype will initiate a file send to a user
</td>

Action

Call User

Add User

Chat with User

View Profile

Leave Voicemail

Send File

Check Your Apple Product Warranty Online

2011-10-01T20:50:13-00:00

If you would like to check the status of your Apple products, enter the serial number below and click ‘Check Warranty’. If you need help finding your serial, take a look at Apple’s official instructions.

How to Report IP or Copyright Infringements on Facebook

2011-10-01T18:45:44-00:00

One task I had to recently complete was cleaning up the Facebook ecosystem for a particular business. Around Facebook there were a few groups, profiles and pages purporting to be the business which needed removing. There are two types of infringement that you can report to Facebook and get actioned, explained below.

Copyrighted Works

For copyrighted works, such as music, movies or images you can have them removed via the Facebook automated DCMA removal form.

Intellectual Property

If you are the legitimate representative of a business you can ask Facebook to remove any unofficial pages, groups or profiles which are purporting to be you. You need to complete the Facebook non-copyright removal form.

2011-09-17T00:00:00-00:00

commercial—
id: 38
title: What is a QR Code?
date: 2011-09-17T15:37:15+00:00

guid: http://beta.dor.ky/2011/09/17/what-is-a-qr-code/
permalink: /what-is-a-qr-code/

This is a post that I’ve constructed using a number of sources (linked at the bottom of this post) and from my own use of QR codes.

Purpose

A QR code (abbreviated from Quick Response code) is a type of matrix barcode (or two-dimensional code) first designed for the automotive industry. More recently, the system has become popular outside of industry due to its fast readability and comparatively large storage capacity. The code consists of black modules arranged in a square pattern on a white background. The information encoded can be made up of any kind of data (e.g. binary, alphanumeric, or Kanji symbols).

QR codes storing addresses and Uniform Resource Locators (URLs) may appear in magazines, on signs, on buses, on business cards, or on almost any object about which users might need information. Users with a camera phone equipped with the correct reader application can scan the image of the QR code to display text, contact information, connect to a wireless network, or open a web page in the telephone’s browser. This act of linking from physical world objects is termed hard-linking or object hyperlinking.

QR codes can be used in Google’s mobile Android operating system via both their own Google Goggles application or 3rd party barcode scanners like ZXing or Kaywa. The browser supports URI redirection, which allows QR codes to send metadata to existing applications on the device. Nokia’s Symbian operating system features a barcode scanner which can read QR codes, while mbarcode is a QR code reader for the Maemo operating system. In the Apple iOS, a QR code reader is not natively included, but more than fifty paid and free apps are available with both scanning capabilities and hard-linking to URI available. With BlackBerry devices, the App World application can natively scan QR codes and load any recognised Web URLs on the device’s Web browser. Following an upcoming update, Windows Phone 7 will be able to scan QR codes through the Bing search app.

History

Created by Toyota subsidiary Denso Wave in 1994, the QR code is one of the most popular types of two-dimensional barcodes. The QR code was designed to allow its contents to be decoded at high speed.

Although initially used for tracking parts in vehicle manufacturing, QR codes as of 2011 are used in a much broader context. Uses extend from commercial tracking to entertainment and from product marketing to in-store product labelling. Many of these applications target toward mobile-phone users (via mobile tagging). Users may receive text, add a vCard contact to their device, open a Uniform Resource Identifier (URI), or compose an e-mail or text message after scanning QR codes. They can generate and print their own QR codes for others to scan and use by visiting one of several paid and free QR code generating sites or apps. Google has a popular API to generate QR codes, and Apps for scanning QR codes can be found on nearly all smartphone devices.

There are several standards in documents covering the physical encoding of QR codes:

October 1997
AIM (Association for Automatic Identification and Mobility) International
```

```
January 1999
JIS X 0510
```

```
June 2000
ISO/IEC 18004:2000 Information technology
Automatic identification and data capture techniques
Bar code symbology
QR code (now withdrawn)
Defines QR code models 1 and 2 symbols.
```

```
September 1, 2006
ISO/IEC 18004:2006 Information technology
Automatic identification and data capture techniques
QR code 2005 bar code symbology specification
Defines QR code 2005 symbols, an extension of QR code model 2. Does not specify how to read QR code model 1 symbols, or require this for compliance.

At the application layer, there is some variation between implementations. NTT DoCoMo has established de facto standards for the encoding of URLs, contact information, and several other data types. The open-source “ZXing” project maintains a list of QR code data types which are discussed in the ‘Embedded Media into QR Codes’ section of this post.

Adoption

The technology has seen frequent use in Japan and South Korea; the United Kingdom is the seventh-largest national consumer of QR codes. In the US, QR Code usage is expanding. During the month of June 2011, according to one study, 14 million mobile users scanned a QR Code or a barcode. 58% of those users scanned a QR or bar code from their home, while 39% scanned from retail stores. 60% of the 14 million users were men between the age of 18-34.

While the adoption of QR codes in some markets has been slow to begin (particularly in markets such as the United States where competing standards such as Data Matrix exist), the technology is gaining some traction in the smartphone market. Many Android, Nokia, Blackberry handsets, and the Nintendo 3DS, come with QR code readers installed. QR reader software is available for most mobile platforms. Moreover there are number of online QR code generators where users can create QR codes for their own needs.

Embedded Media into QR Codes

The actual idea of embedded media into a QR is a bit of a myth. QR codes can only store alphanumeric characters and a selection of symbols. You can invoke different types of actions by using URL/protocol schemes which are defined by each QR reader. The following list details the types of URL schemes that you can use.

Contact Information

QR codes can contain contact information so someone can easily scan a QR code, view your contact details, and add you on their phone. You can input your name, phone number, e-mail, address, website, memo, and more. I've commonly seen these used on business cards, for instance. You give someone a business card, they see a QR code on the back, scan it with their phone, and easily add your contact info to their phone.

Calendar Event:

If you have an event you want to promote, you can create a QR code containing info for that event. QR codes containing event info can contain event title, start and end date/time, time zone, location, and description. This could work well on an event flyer or possibly even on a website promoting.

E-mail Address:

Nice and simply, eh? A QR code can contain your e-mail address so someone can scan the code, see your e-mail, and then open an e-mail on their phones. If your call to action is mostly to have someone e-mail you, this would be great.

Phone Number:

Maybe e-mail isn't immediate enough and you want someone to call. Link them up to a phone number.

Geolocation:

If you have an event you want to promote, you might want to stick a QR code linking someone to a Google Maps location. This will allow someone to scan your QR code and get directions so they don't have to manually type in an address. Although some may prefer to type it in, it doesn't hurt to give them another option.

SMS:

QR codes can populate a text message with a number and message. You can have your QR code send you a text saying, &#8220;Tell me more about XYZ&#8221; for instance. This is great when paired with text message marketing.

Text:

You can also just have a sentence or a paragraph of text. This could be fun for having some type of QR code based game where you can leave hints in QR codes.

Wifi Network:

Do you hate telling someone a long WEP wireless key that's a pain to type out on a mobile phone? Set it up so someone can scan a QR code and automatically configure wifi on their phones.

URL:

Don't think little of this 3-letter data type. This is where the possibilities become endless. You can use a link that takes someone to your Facebook fan page or Twitter profile. You can also link someone to a YouTube video. Or maybe you want someone to pay for something via PayPal.

Did I mention all of these services have mobile-friendly versions of their website? That is, if you link someone to your Twitter page they'll be redirected to a version that's built specifically for their smartphone? Don't forget to make sure the link goes to a website that's optimised for mobile devices.

Licensing

The use of QR codes is free of any license. The QR code is clearly defined and published as an ISO standard. Denso Wave owns the patent rights on QR codes, but has chosen not to exercise them. In the US, the granted QR code patent is US5726435. In Japan it is JP2938338. In Germany it is DE69518098. (The European Patent Office granted patent EP0672994 to Denso Wave, but Denso only “nationalised” the patent grant in Germany.) The term QR code itself is a registered trademark of Denso Wave Incorporated.

Risks of using QR Codes

Malicious QR Codes combined with a permissive Reader can put a computer’s contents and user’s privacy at risk. QR Codes intentionally obscure and compress their contents and intent to humans. QR codes are easily created and may be affixed over legitimate QR Codes. On a smartphone, the Reader’s many permissions may allow use of the camera, full internet access, read/write contact data, GPS, read browser history, read/write local storage, and global system changes.

Risks include linking to dangerous websites with browser exploits, enabling the microphone/camera/GPS and then streaming those feeds to a remote server, exfiltrating sensitive data (passwords, files, contacts, transactions), sending email/SMS/IM messages or DDOS packets as part of a botnet, corrupting your privacy settings, stealing your identity, and even containing malicious logic themselves such as JavaScript or a virus. These actions may occur in the background whereas the user only sees the Reader open a harmless webpage.

Commercial Uses

The use of commercial QR codes is increasingly greatly. The recent Red Hot Chilli Peppers album adverts for the album “I’m With You” had a creative QR code within the posters as Catalina Cadena notes in her blog post.

Data capacity

The amount of data that can be stored in the QR code depends on the character set, version and error correction level.

Error Correction

Codewords are 8 bits long and use the Reed–Solomon error correction algorithm with four error correction levels. The higher the error correction level, the less storage capacity. At the highest error correction level it is possible to create artistic QR codes that still scan correctly, but contain intentional errors to make them more readable or attractive to the human eye, as well as to incorporate colours, logos and other features into the QR code block. While the exact number of errors that can be corrected depends on the size of the symbol and the location of the errors, the following table lists the approximate error correction capability at each of the four levels:

Level L — 7% of codewords can be restored
Level M — 15% of codewords can be restored
Level Q — 25% of codewords can be restored
Level H — 30% of codewords can be restored

Uses of QR

Business Cards.
Stick a QR code on the back of your business card and have it link to your phone number, send a text message, link to your website, or even take someone to a Youtube video.
Flyers.
Handing out flyers door-to-door? Or maybe you want to spread the news about an event in a new and innovative way? Slap a QR code on it and link to an event page or a video describing more about the event. Check out how one clothing retailer is doing this. You can also add social media icons so people can share.
Billboards.
This one way communication channel just became interactive. Calvin Klein used QR codes on billboards in New York and Los Angeles with their Uncensored campaign. It linked to a page with a mobile video and Twitter/Facebook share icons. Simple? You bet. Learn more about it.
Comics.
That’s right, QR codes have been used in comics. It was done with Donald Duck for kids!
Education.
Use QR codes next to math problems in math books that pull up a video that teaches students how to complete certain problems. Let’s just hope they scan the QR code after they answered it! I picked up this idea on Youtube.
Articles in Magazines/Newspaper.
Magazine publishers can use QR codes in articles that link to more info on the article, a video, or even a discussion board. Wait. Stop. Think about that. Another one-way communication channel that is now engaging. Imagine someone reading an article, wanting to voice their opinion and leave a comment like they would on a website blog, scanning a QR code, and then leaving comments on an article they found in the paper medium. Talk about in-the-moment engagement. Simply fascinating.
For Sale Signs on Homes.
Real Estate, anyone? Print a QR code on a for sale sign on a home that connects the mobile user to a mobile website that shows them more info on the home and includes a video with a walk-through of the home (which they wouldn’t otherwise be able to see). Or just link to a Youtube video. Talk about point-of-experience!
Comment Cards.
Tired of paper comment cards? Want to reduce error and capture feedback at the point-of-experience? Or maybe there is a trust issue with the manager and you think he’s trashing some bad comment cards? Either way, mobile web-based comment cards allow for convenience and meet the consumer where there are. Comments can be e-mailed to someone and added to an excel file if need be. Theoretically, someone can file a complaint and a manager (as well as corporate, if needed) can receive the complaint via e-mail and respond to it before the customer has even left the building. Talk about real-time.
Presentations.
More and more I’ve seen less of the traditional speaker presentations. Audiences want to engage during presentations and throw in their two cents, just like they do on Twitter. The unGEEKED e’lite conference is built on this idea. One way to engage the audience is to stick a QR code in a presentation. Maybe the audience can scan it and be put on a mailing list, ask a question, or give their input on the presentation at the end, all via the mobile web. This would work best at conferences where early adopters are present.
Clothing.
Want to be different? How about you stick a QR code on a shirt. How’s that for standing out against the crowd? Or clothing companies can use QR codes on tags to display colour, style, size, and more about that item.
Television.
This is extremely fascinating. Again, a one way communication channel… is now interactive. QR codes on TV allows for viewers to answer quizzes, sign up for a mailing list, find a Facebook fan page, and more. Although executed poorly, FOX tried this.
Books.
How about a QR code at the end of each chapter, which links to a discussion board where readers can share comments on a chapter. Amazing. Or maybe you want to describe a concept with digital media. Video, anyone? Or maybe the author wants to link you to a page with errata.
Outdoor Games.
Ever participate in a scavenger hunt? With QR codes, you can leave digital tips for people. How about a video hint or video riddle?
Direct Mail.
QR codes on direct mail allows for people to scan and learn more about a product or service.
Instruction Manual.
With some products, a paper manual isn’t enough. People need a video to help visualise how things go together. Or maybe you just want to give that personal feeling and talk about why it’s awesome.
Interactive Menus.
In restaurants, consumers can scan QR codes in the menu and watch a video on how that dish was prepared, see the ingredients, learn more about the executive chef with an interview, or even the staff. Stop. Think about that personal connection for a minute. Think about the story that someone can go tell their friends. They came to order food, but their experience was heightened because they were also able to walk away with a recipe for their favorite dish (or something more). That is remarkable, as Seth Godin says. Again, the possibilities are endless. What’s important is the creative process and brainstorming.
Cereal Box.
Cereal boxes already have games on them, but how about games on a smartphone? Or maybe you want to have a “story of the day” or talk about what the brand is doing. This is a completely new way to engage with consumers.
Products.
QR codes can be used on products to connect real life to the digital world. Maybe you want to build awareness for a Facebook fan page or a Twitter account.
Facebook Like.
A service called Likify allows someone to do exactly that: like your Facebook fan page. Learn more about them.
Call/E-mail Us.
Simple enough? Have a QR code that brings up your phone number so someone can easily scan and call you or send you an e-mail.
Coupons.
Use a QR code to link to a special coupon that is shareable via social media. It can be tracked and turned off at any time. Goodbye paper coupons! (Did I mention this is environmentally friendly, too?)
Name Tags.
Although the idea of someone coming up and scanning your name tag at a conference sounds kind of funny, it can definitely help differentiate you and be a great conversation starter. There is a standard for storing contact info on QR codes. Use quiQR to create your virtual contact card today so someone can easily add you as a contact on their phone.
Website Contact Us.
If you already cater to your mobile audience, you can use QR codes on your contact us page. Maybe you want to link them to your mobile resource and remind them to bookmark it so they can use it on the go. Or maybe you want to make it so they can easily scan a QR code and call you.
Conferences.
Conferences can use QR codes for speaker feedback or conference feedback and evaluation. Attendees can share their thoughts via their mobile phone, and not a paper card with hard-to-read handwriting.
Event Tickets.
On a printed ticket, QR codes can be used to link to a video introduction to the event, or maybe link to a free MP3 download for a band. This is also good for tracking tickets, which Eventbrite does.
Window Displays.
Use QR codes on windows displays to give those window shoppers one more thing to look at (and talk about!). It can link to a video with someone talking about the product (very personal), or maybe you want to highlight certain features about the product.
Classified Ads.
Selling a car? Hiring someone? Link to a personal video on Youtube which has you expanding on the opportunity or telling people why it’s a good one.

Variants of QR Codes

Micro QR code is a smaller version of the QR code standard for applications with less ability to handle large scans. There are different forms of Micro QR codes as well. The highest of these can hold 35 numeric characters. Standard QR code is the QR code standard for applications that possess the ability to handle large scans. A standard QR code can contain up to 7089 characters, though not all QR readers can accept that much data.

Although encrypted QR codes are not very common, there are a few implementations. An Android app, for example, manages encryption and decryption of QR codes using a secure AES 128 algorithm. A special form of secure quick response codes utilising public key encryption are SKS codes which have identical appearance to normal QR codes. SKS codes were designed for one-time-use and are created by encrypting a precursor hardware security token such as the CID written in the ROM of a microSD card, applying a time stamp and displaying the code on for example the high resolution screen of a mobile handset. SKS codes are closely analogous to One-time pad which are highly secure and can therefore be used for applications demanding high security such as mobile financial transactions. SKS codes in fact combine the CID and GMT with other proprietary data related to the transaction, an assembly of datum which never occurs again and which is discarded after being decrypted and processed on the transaction server. SKS codes were first developed in 2010 by Japanese company Yodo KK and are patent pending.

Links/Credits

Expose Email Addresses via Twitter API

2011-09-10T16:39:14-00:00

One of the most popular posts that I wrote previously for my blog was regarding the Twitter API and email addresses. That post has since been permanently archived and is no longer available It seems as though this topic is very popular among new developers to the Twitter ecosystem. In this post, I’m going to revisit that topic once again.

How do I get the email address of a user using the Twitter API?

This is a very simple question to answer. You can’t.

Why are email addresses not available via the Twitter API?

There is not a single reason why Twitter should hand over the email addresses of its users to anyone. Any application that you develop should explicitly ask the user the input their email address, clearly stating why you believe you need it.

User addresses are private, just like telephone numbers. The only need for email addresses to be exposed via the API is for spam harvesting which is a big no-no.

I want to keep my application users informed about changes

This is a reasonable thought, but consider the viewpoint of your users. The will follow the account associated with your application for news and updates. Users will also expect to find news, updates and important content via your application.

Will Twitter ever open up the API to disclose email addresses?

No, and that’s a good thing.

Fixing a Rattling Garmin eTrex Vista HCx

2010-12-31T12:27:48-00:00

The eTrex Vista HCx is a fantastic little GPSr. The one that I currently have is from Matthew who loaned it earlier this year when our Dakota malfunctioned. After being out in the Roaches, the eTrex suffered an impact injury and began to rattle. Fortunately this was easy to fix as it was only a cover of the speaker inside the GPSr.

To begin, remove the batteries and battery cover. Now remove the black rubber casing band around the GPSr. When you’ve removed this completely you will see some clear plastic tape of which there are three layers. Remove each clear tape label in turn and then carefully keep them to the side as you’ll need them afterwards. You will then have your GPSr open as shown in the photo below. Unclip the top from the bottom gently as this will make reattaching the broken piece easier.

You can now see the piece that has broken off (if you’ve broken the same piece as me that is). If you’re lucky you will be able to clip it back into place. Failing that then reattach it and super-glue into place but be sure of leave two sides open so the speak still works correctly. The image below shows the speaker without its case attached.

Carefully put your eTrex back together and it’ll be as good as new. You can also use these instructions for opening your eTrex for anything else you may need to.

How to Install OpenStreetMap Maps onto a Garmin GPSr

2010-11-12T22:41:50-00:00

Garmin provide many types of maps for their devices but each costs quite a large sum of money. Thankfully in recent months this process has become a lot easier for the end user to do. This short guide will explain where to find the correct map file and how to install it using a Dakota 10 GPSr, a Mac and Garmin"s free software.

OpenStreetMap

The OpenStreetMap project provides map data for a large amount of the world for free. Even though the project is community edited – its still an excellent free map to use on your portable GPSr device. If you take a look at the list of maps available on the OSM site (located here for the UK) you will see how many different people are providing packages/data files that can be used with Garmin devices.

Find Your New Map

The choice I made for UK maps with contours and routing is the maps created by user “Talkytoaster” and can be found at his website. At the present time the latest file is “101105” and the filename is “101105-UK+Contours-Routable.gmapi.zip” which will be a file compatible with Garmin"s MapInstall software on the Mac.

Install using Garmin MapInstall

Once you have got your file downloaded (which is probably a few hundred MB) then unzip the file you downloaded. You can plug in your GPSr and run the Garmin MapInstall software on your Mac. Follow the process on-screen and after a short time it’ll be done. Reboot your GPSr and you’ll notice that while its loading at boot time you’ll have a piece of text that says “OPENSTREETMAP”.

Why GeoCaching.com Needs an API

2010-10-15T21:25:26-00:00

A lot of what I talk about these days seems to be Geocaching related. The concept of Geocaching has long been around and at present the biggest central location of cache information is at geocaching.com which is owned by Groundspeak. The website, applications and tools are pretty average to use and no real changes have been made to the way that users interact with the site for a long time. The iPhone application is clunky at best and is a difficult to use at times. The reliability of the application over mobile connections is also questionable. All of the caching data is held by Groundspeak and although premium members can retrieve data via Pocket Queries (multiple caches in a single GPX/LOC file) and single file downloads (one cache per GPX/LOC).

There is no public API for developers or users to use. There is no systematic way other than scraping to fetch cache data – and this is against the terms of the site and services. There are many good reasons why Groundspeak should introduce a public developer API. It makes no difference if it is licensed and paid for, developers will still use it and it would boost the bottom line for Groundspeak.

Uses of an API

By opening up the data to developers, better applications can be created and users get choice. Choice is a very good thing. You only need to look at the enormous amount of Twitter applications that have made the ecosystem flourish and people can decide to use an application that provides them with the features that they want and less of what they don’t want.

Pushing Innovation

This fuelling of innovation can only help Groundspeak increase its revenues and expand Geocaching to a bigger audience. The more people that are using the API, site and products can only enhance what Groundspeak have to offer. Using the example of Twitter clients again, products that have implemented good ideas and functions often get copied: as the famous quotation goes “imitation is the highest form of flattery”. The problem with this is that at the present time Groundspeak are merely stagnating Geocaching for developers and in turn the end user.

Implementation

Any public facing API needs to be controlled in access and usage. This is to prevent aggressive over use and people not abiding by the terms of use. Leaning towards Facebook and Twitter’s APIs who both have very good usage policies and methods of dealing with over use by rate limiting, you can see effective throughput can be efficiently handled and scaled.

Free Use or Premium

The other side of providing the API is the cost of covering running the service. Access to it could be funded by providing to premium users only. If access if provided to premium users only, even without an increase in price the bottom line for Groundspeak would improve and a lot more users would pump money into the ecosystem by purchasing third party applications that would require premium memberships.

GCE:Standard

Any implementation of an API would likely provide either XML or JSON. An extension of this could be a new multi purpose format for cache information exchange, perhaps a new GeoCacheExchange:standard.

T-Mobile Customers Can Now Use Orange Too

2010-10-12T17:19:07-00:00

Earlier this year T-Mobile announced that they were merging with Orange. This didn’t matter too much to most at the time but its changed a little now. For a while they’ve been testing network sharing and its now live for customers to use. If you’re on T-Mobile you need to first text ‘Yes’ to 2121. This will switch your account to use both networks and whichever has a signal. You’ll get a text back saying that your account has been changed. Remember that at present if you’re a T-Mobile customer using the Orange network you only get 2G, not 3G so things like YouTube won’t work (unless you’re jailbroken and forcing it to).

I’ve tested it myself and it works flawlessly. If you’re an iPhone user you will need to make a few change to your settings. Goto Settings.app, click ‘General’ then ‘Network’ and turn on Data Roaming. If you want to force your phone into using Orange, click ‘Carrier’ and change to Orange (or T-Mobile Orange if its called that for you).

Twitter API and Email Addresses

2010-10-02T22:17:00-00:00

One thing that appears to be a recurring theme on the Twitter API development mailing list and in the IRC channel of Freenode is the question of getting access to a users email address. I’ve always found this frustrating and have never seen a good reason why a users email address should be given out via any API method. The following list contains the reasons that are most often given when you ask why an email address is desired:

I’d Like to Contact the users of my application:

See below.

Auto-Notices & Spam:

Not a chance.

It really is that simple. If you need a users email address then provide a settings page and ask them to enter it. Don’t think you can systematically fetch it from the API because you can’t, more to the point: I don’t think you’ll ever be able to.

And that’s a good thing.

Open Data Results

2010-10-01T20:40:23-00:00

I was asked earlier this year to take part in an Open Data survey to see how data was being used. The survey was performed by Tim Davies. You can find the report here or a summary here.

A Guide to Geocaching

2010-09-06T10:13:18-00:00

As many of the regular readers of this site and my Twitter account will know, I’ve very much integrated GeoCaching into my daily life. Just like pringles you won’t be able to stop once you’ve started. GeoCaching is a very simple premise that is similar in principle to the old idea of letter-boxing. You can sign up at geocaching.com for an account an begin looking for GeoCaches in your area straight away. You will need some form of GPS receiver such as a Garmin. This article provides a little information on how to find a GeoCache and the best ways to hide your own caches.

Finding a GeoCache

When attempting to find a GeoCache there is no particular set of rules that you need to adhere to. The following is a list of how I’d recommend you start looking for a GeoCache.

Look for something out of place, or something placed deliberately as its extremely difficult to make a hide look natural.
Be aware of the container you're looking for, look for a gap or hole that the container could fit in.
Caches are often covered in leaves, rocks, sticks or a camouflage bag.
If you're new to GeoCaching, then read the hint provided, look at the photos posted and read recent logs of the cache to see what other people have said when looking around for it.
Have a little patience. Sometimes it can take a while to find a cache, even experienced GeoCaches will have times when they don't find a cache quickly.
Try to be discreet. You're often in public areas and no one wants caches to get trashed/muggled. At times, you will need to use stealth.

How to Hide a GeoCache

After you’ve made your first finds you’ll have a better understanding of what makes a great GeoCache. I usually recommend that people consider hiding caches after they’ve made 50 or so finds as you have a better knowledge of the sport. The list below provides some tips on placing a GeoCache and things to avoid doing.

Cross check your co-ordinates with other devices. iPhones and smart-phones do *not* provide good co-ordinates under any form of tree cover. If you have difficulty in getting co-ordinate accuracy, please note this in your cache listing. People won't mind, they just want to know the right areas to search and finders are happy to post their co-ordinates.
Make your cache hide as natural as possible. If you're using rocks at the base of a tree, or sticks in a rock formation then it will be too easy for the GeoCachers to find.
Holes make great hiding places.
If there is more than one way to approach your cache, make sure you've hidden it from all approaches.
If you need to use a lot of camouflage then consider a different location.
If its on the bank of a waterway, use “East Bank” for descriptions and hints. Using “Its on the other side” is almost useless in these situations.
Try to get permission before placing your cache. Places like Peak District, National Park, Forestry Commission are all happy to give permission for caches. They just like to know where they are.
Try to place caches in areas of interest that people will enjoy seeing. Would you like to go here yourself?
Remember, people love a challenge. Be inventive.
Don't forget that you need to maintain a cache. If for some reason you can no longer maintain it, say so and usually someone is happy to adopt it.

Other Tips

Wear long trousers and take gloves as you'll most likely encounter nettles and hawthorns.
Always take a drink/bag of crisps with you, you'll nearly always be out for longer than you expect.

If you have any other tips or comments then please add them in the comment section for others to read.

Twitter API Oauth Update Status

2010-08-21T21:42:27-00:00

This PHP script will post an update to Twitter using OAuth. You can download the complete script and library here.

The Actual Script

Running the Script

The End Product

Check Apple Device Warranty

2010-08-17T11:49:36-00:00

You can get quick information about any of your Apple devices by using their self service warranty information page, located here. For serial numbers from iPhones and iPads it’ll show supported networks, current warranty status and a little more. Its also useful for checking legitimacy of serial numbers.

Parsing GeoCache GPX with PHP

2010-08-13T16:10:16-00:00

While working on the new cache pages for this site I came across the interesting task of parsing GPX files with PHP. SimpleXML is quite easy to use and with the children() function you can use namespace extensions – for example with GeoCaching exported GPX files. You can get an awful lot more information from a GeoCache GPX than with standard XML parsing.

The following is an example of a GPX file from the Geocaching site:

Now, as you can see there is a large amount of data contained within this file. Using PHP and SimpleXML you can extract this data quite easily, by doing something similar to this:

Local Businesses Should Harness Locality Services

2010-05-07T16:52:41-00:00

There is an emerging trend that local business won’t be able to ignore for too long. At the moment, all the rage with mobile devices seems to be location based services and games with Gowalla, FourSquare and Yelp being the primary platforms although Facebook is to launch its own version shortly which may comprise of an amalgamation of all three. These platforms reward users with points, prizes and badges the more they ‘check in’. I haven’t had chance to test out Yelp more yet, so I’ll exclude that from the results below.

There are a few subtle differences between the two platforms, which I’ll detail below. I’ve found that most people seem to use both Gowalla and FourSquare. At the moment, I’m leaning more towards being a Gowalla permanent user. I’ll outline a few differences with the platforms below.

<th class="matrixColumn">
  <h5>
    Foursquare
  </h5>
  
  <p>
    <img src="/media/icon/other/foursquare.png" alt=" " /></th> 
    
    <th class="matrixColumn">
      <h5>
        Gowalla
      </h5>
      
      <p>
        <img src="/media/icon/other/gowalla.png" alt=" " /></th> 
        
        <th class="matrixColumn">
          <h5>
            Brightkite
          </h5>
          
          <p>
            <img src="/media/icon/other/brightkite.png" alt=" " /></th> </tr> 
            
            <tr>
              <td class="matrixItem">
                Marking as Present
              </td>
              
              <td class="matrixOdd">
                Check In
              </td>
              
              <td class="matrixEven">
                Check In
              </td>
              
              <td class="matrixOdd">
                Check In
              </td>
            </tr>
            
            <tr>
              <td class="matrixItem">
                Most Frequent Visitor
              </td>
              
              <td class="matrixOdd">
                Mayor
              </td>
              
              <td class="matrixEven">
                Leader
              </td>
              
              <td class="matrixOdd">
                TBC
              </td>
            </tr>
            
            <tr>
              <td class="matrixItem">
                Media Uploads
              </td>
              
              <td class="matrixOdd checkNone">
              </td>
              
              <td class="matrixEven checkMark">
              </td>
              
              <td class="matrixOdd checkMark">
              </td>
            </tr>
            
            <tr>
              <td class="matrixItem">
                API
              </td>
              
              <td class="matrixOdd checkMark">
              </td>
              
              <td class="matrixEven checkMark">
              </td>
              
              <td class="matrixOdd checkMark">
              </td>
            </tr>
            
            <tr>
              <td class="matrixItem">
                Userbase
              </td>
              
              <td class="matrixOdd">
                TBC
              </td>
              
              <td class="matrixEven">
                TBC
              </td>
              
              <td class="matrixOdd">
                TBC
              </td>
            </tr>
            
            <tr>
              <td class="matrixItem last">
                Push Notices
              </td>
              
              <td class="matrixOdd checkNone last">
              </td>
              
              <td class="matrixEven checkMark last">
              </td>
              
              <td class="matrixOdd last">
                TBC
              </td>
            </tr></tbody> </table> 
            
            <p>
               
            </p>
            
            <p>
              The way in which people interact with business has changed. The entire customer service model is slowly changing and those who do not update their methods and ways will be left behind. In the US, the adoption of FourSquare and Gowalla has been picking up rapidly whereas in the UK its seen a slower adoption pace. One thing is for definite though, if you decide to provide offers to your customers/users of these services then you will drive traffic to your business and promote a better reputation with the digitalrati of the modern age.
            </p>
            
            <p>
              This can provide to be key in the way in which your business is viewed. Trendy coffee bars in town centres are interested in seeing how they can use Gowalla and Foursquare to promote their business better. Promoting via one of these online services is like word-of-mouth on steroids. It’ll quickly get around the services and attract new people when they see their friends at your business.
            </p>
            
            <p>
              Cafe's in the larger cities have already started offering free coffee and a muffin for the Mayor of their location. Believe me, no one will refuse a free muffin.
            </p>

Feature / Terminology

Unified Reward Point System

2010-03-26T14:47:55-00:00

The system of rewarding customers with points, cards or discounts is a fairly standard practice amongst today’s modern business. This reward model is based around rewarding for purchases of goods and services and then allow these customers to choose a gift in return to reward their loyalty. The existing implementation are this reward network is singular in nature except for organisations such as Sainsbury’s nectar.

I believe that modern customers would be more beneficial to a system that was shared between businesses and then allowed the customer to take reward points to another company should they wish to do so. Providing the spread of reward was equal or near to equal over all participating organisations, then this model would be extremely beneficial to the customer who may for example not wish to have a gift from shop A, instead choosing a gift from shop B.

The new age innovations that customers have access to such as the Internet, mobile phones and smart phones provide rich platforms for interaction with business. Devices such as mobile phones or smart phones that have Bluetooth could be registered against a user record and then through transmitters in store, promotional adverts or messages could be delivered via OBEX push when the user is nearby, as shown in the diagram below.

The transmitters shown above would be looking for known Bluetooth devices. Of course this would only happen if the handsets were configured to allow OBEX items to be pushed. Alternatively, an email could be sent to the user (taken at opt-in) with discounts coupons for that day. Bluetooth’s proximity would allow tracking up to around 10 metres for most handsets which would be close enough for interaction. This would then drive customers to pursue interesting offers and lead to more sales.

In the diagram below, relationships are displayed in colours so that it makes it easier to indicate what they represent. The yellow arrows show the relationships between a customer or group of customers and the businesses that they frequent. The purple arrows show the intranet-based or Internet-based link up to a central system. The blue arrow and read our indicate a link between the customer of being at home or on a smart device and interacting with the central system. This could be to engage in choosing a reward gift or simply updating their account details.

This system would also allow for flexibility and grouping users so that they may work together and obtain larger gifts. The biggest challenge in implementing a better, more advanced system would be gaining the participation of the larger organisations from which people choose to purchase their goods from. I do honestly believe that gap could be bridged then it would not be too far from the future to implement such a new system.

Platform vs Web Site

2010-03-24T15:47:21-00:00

Over the past six years we’ve seen major developments around the world in online services and computing. Traditional web sites such as Google, Amazon and eBay have now transformed into multinational platforms offering more than just a site to use. All of the services I’ve just named still provide their main web site, but isn’t it about time we started referring to them as platforms rather than sites. In my opinion, it is.

You can do so much more than simply search with Google or buy with Amazon and eBay. I don’t see the naming and references changing any time soon, but this is a thought I’d like to see come to fruition. The traditional roots of each of these large platforms can never be forgotten but through their own research, development and improvements they’re providing excellent services for the rest of the world to use.

In the more recent history, Twitter and Facebook have provided platforms for people to develop with which in turn created a boom for both enterprises. Its nice to see Facebook referring to their infrastructure and service as a platform, rather than a site. Perhaps others can follow suit, but I doubt it.

Apple OSX Temporary Fix for Dead Ethernet Adapter

2010-03-20T11:07:43-00:00

Really, robots.txt is There for a Reason

2010-03-06T14:31:08-00:00

Quite a while back now in Internet history a system called ‘robots.txt’ was introduced to control the content that web crawlers and services are permissible to crawl. This file details a map which denotes what crawler/bot is allowed to crawl a site. At tweekly.fm I deal with a lot of user traffic and the complicated nature of our queries means that its quite intensive to generate the user pages.

There are multiple levels of caching and memcache is there too but even under peak times we still have to serve pages to users. The most infuriating thing is during the period in which we send tweets out each day for users, we get bombarded with requests from crawlers and bots. This slows both our service and our processing down but if the crawlers in question were to honour robots.txt as they should do then this wouldn’t even be an issue. It wouldn’t even be too bad if requests were staggered out over a period of time.

The worst offenders at first glance are TweetMeme, Radian6, PostRank, TwittUrls, MetaURI, Twingly, Page-Store & Chainn.

Tweekly.fm Featured in a Book

2010-03-05T12:01:14-00:00

Sometime last week I was talking to Simon about Tweekly.fm and he informed me that Tweekly.fm was going to be feature in a book that’s being written by Garin Kilpatrick. Garin’s book will document tools to be used with Twitter and we’re happy that we’re going to be featured. Its nice to see that the service being provided to end users is a good one and that we’re being recognised as a useful tool.

Caching & ADOdb for Tweekly.fm

2010-02-26T21:59:24-00:00

The best thing about building a project that gets popular is the same as the worst things about a project that grows rapidly. Tweekly.fm is currently gaining 1,500 users per month most of which decide to publish on a Sunday. This increases the load and run time of the delivery engine that pushes tweets outbound. The other side of an increased user base is that user pages get more popular. Although this is a good thing overall and shows our popularity – it also has its downsides.

The queries to show simple people and shared by counts are expensive to run. We’re indexing just short of 25,000 user records and running the queries in the original way brought the servers down to a sluggish speed. My response to this was to rewrite the entire user pages and introduce caching on a couple of levels.

The diagram below shows the original connections between our servers and our users. As the diagram shows, there was no caching present through the system. This presented problems under load and ended up with the user pages of the site being temporarily postponed.

After a little research and crunching of numbers I came down to the conclusion that caching points at both the database server and the website (abstraction layer). Implementing the ADOdb layer underneath the existing database links has proven to be a great success. Its boosted response times over 300%. The next diagram shows the new caching points added the the system.

Point A is the abstraction layer at the site level. At peak times, the queries issued here are cached for an hour. This scales back to 30 minutes at low load times. Point B is the connection between the web server and database server. Queries are cached at the database constantly via the MySQL Query Cache. Finally Point C is the connection to the Twitter API. All calls are cached for 30 minutes where possible, but due to the nature of the API and service in general I’m not currently caching too much here. What is not shown on the diagram is the connection to the last.fm API because this data is automatically cached and updated when needed.

Effects of Changes for Users

The effects of these changes will be noticeable by all users because the front-end of the site will be a lot snappier and responses should be quicker too. Hopefully this is just one of many changes that can improve the system for everyone.

Social Traffic Alerting

2010-02-25T10:14:09-00:00

I spent some time yesterday extending the StokeTraffic platform to include support for eye witness reports provided by users of the service. This now means that the service will include real time eye witness reports of problems and conditions of the roads in and around Stoke on Trent. All users are invited to participate in this extension of the service by using the new feature. If you’re travelling in or around Stoke on Trent and spot a problem then you can alert us by posting a tweet similar to the following:

@stoketraffic rep A500 Extremely busy due to congestion

If your report is accepted it’ll be posted live within a few moments. Please be aware that reports are limited to 100 characters and there are measures in place to prevent abuse to this service. I’m looking for suggestions, features and also of any issues. Get in contact with me here.

Geocaching.com Storing Plaintext Passwords

2010-02-04T23:06:17-00:00

I completely found myself in shock earlier today when I had to use the ‘forgotten password’ feature on geocaching.com. To my complete amazement, they emailed my password back to me in plaintext. Now, bearing in mind that most of their site is basic in features I’m sure they haven’t generated their own two way encryption so that means they’re storing passwords plaintext which in this modern age is suicidal.

I simply couldn’t believe it, I hope they can prove me wrong but I’m almost sure they storing them plaintext. Please, someone from geocaching.com contact me and tell me you have some space age encryption going. Until then, I’m changing my password to something retarded.

LaFonera as a Wifi Bridge/Repeater

2010-01-17T13:47:08-00:00

This is a screenshot of the bandwidth usage for the Fonera router I flashed last week. Everything seems to be working fine. A couple of PDF documents that I found useful can be downloaded here and here.

Last.fm Auto-Tweeting

2010-01-17T03:12:43-00:00

This has been moved to http://tweekly.fm as I’ve rewritten and redeveloped the tweekly system.

Also, there is a Facebook version here at laststat.us

~~I've just finished the beta of my autotweeting~~ ~~last.fm~~ ~~statistics system. It'll take your last.fm top artists and publish them to Twitter on a day of your choosing.You can sign up for the~~ ~~beta here~~.

If you have any problems or encounter errors while using the system then please contact me as soon as you can do at scott@dor.ky.

Hacking a Fonera Router

2010-01-15T20:04:30-00:00

A few people have asked about more details for the wireless bridge I’ve set up within my local network. You can find more information about it here. Its a pretty simple process and you end up with a router running DDWRT.

I’ve got it set up to bridge the ethernet port into the main wireless network here.

Google Latitude API Non-Existance

2009-12-10T19:23:24-00:00

I’ve just written a request for more information from Google as I’m astounded to find no accessible API for Latitude. I find that completely shocking on their part.

I’ve looked on the AppSpot site and here in the group and I’ve not seen a definitive answer as such to this question. Is there an accessible API to update my location on Latitude? I already have my location via GPS elsewhere and would very much like to bridge this over the Latitude and update my location there.

I see that at present, this is not possible through existing Latitude services, however I’m interested in knowing whether or not Google plans to introduce this in future and if so – a possible time frame for doing so. I have an application that I would like to use this functionality in.

Scott.

You can see the actual post and follow it here.

Posts - Scott Robinson · @ssxio

How to Detect Service Versions on Adobe Commerce Cloud

Valkey

MariaDB

RabbitMQ

Why this matters

Blocking SessionReaper and PolyShell File Access with Fastly on Magento/Adobe Commerce

Blocking access at the edge with Fastly

Edge-level blocking over application-level blocking

Important Note

Further reading

Fixing Magento's Telephone Validation to Allow Dots and Slashes

Creating URL Redirects When Updating Product URL Keys via the Magento API

The API request

Why this matters for bulk operations

Verifying the redirect was created

Fixing Broken Product Relationships in Adobe Commerce/Magento 2

Finding the problem

Checking for orphans

Cleaning up

Disable Chrome AI Mode in the Omnibox

Claude Code - How to Revert Back to Opus 4.5 from Opus 4.6

Signing Git Commits with SSH

Adobe Commerce Isolated Security Patches Changelog

Adobe Commerce Deployments Failing - Composer block-insecure in Effect for 2.7/2.8

Nginx client_header_buffer_size - Debugging Cloudflare-related 400 Errors for Magento/Adobe Commerce HTTP Requests

Adobe Commerce/Magento 2 Concurrent Import Process Interference Issue

Reproduction

How to List/Change/Activate Different Service Versions on Fastly for Adobe Commerce

Flush Fastly CDN via CLI/Command Line for Adobe Commerce Projects

Adobe Commerce Deployment issue 'does not support Valkey backend model'

Adobe Commerce B2B upgrade issue 'REGEXP_LIKE does not exist'

Laravel 12 $request->getContentType() does not exist

Set Website/Store Assignment via Magento 2 API

Can't place the order error for Paypal Express (when Stripe is also installed) with Magento 2

Hyva Not Generating app/etc/hyva-themes.json on Deployment for Magento 2.4.7

Klarna Checkout Won't Install on Magento 2.4.4

How to Re-run a Composer Script After install/create-project

Magento 2.4.4 Admin Login Redirect Loop/Not Working

Create Repeating Google Calendar Event for Last Working Day of Month

Create Repeating Google Calendar Event for Last Specific Day of Month

Create Repeating Google Calendar Event for Last Day of Month

Using logrotate as a non-root user

How to Install PHP 7.2 (and 7.1, 7.4 etc) on MacOS Monterey

Easily Create Windows 10/8/7 (or any other OS) Bootable USB Drives

Wordpress Docker - How to set memory_limit, max_upload_size and other resource limits

How to Stop Yourself Bricking Your Mac - Disable Secure Boot

Magento 2 - How to delete all special prices

macOS Big Sur - Installing pip for Python

A quick way to truncate files on Mac and Linux systems

Show user groups on Linux

A neat app for team safety checkins and announcements

Keeping Your Privacy: How to Set-up Your Own VPN Using Cloud Providers

Our VPN Choice

Select a Cloud Provider

Pick Your Subdomain and Configure DNS

Streisand Installation

Device Configuration

Follow on:

Keeping Your Privacy: Face Blurring & EXIF Data

Blurred Lines

Be Wary of

One Last Thing - EXIF

Introducing a Series: Keeping Your Privacy

How to Install and Configure Docker on Ubuntu 20.04

How to Configure and Use Hardware Yubikey with Mac OS SSH

…and without the hardware key?

Using Additional Configuration Files with Laravel & Lumen

Git reflog can help you restore lost work

How-to set downloaded filename for S3 pre-signed URLs in PHP

How to Zip and Encrypt a file on a Mac

Strong Customer Authentication (SCA)

Further Reading

PHP 5.6 Vagrant Box for Legacy Projects

Firefox Send

Nginx acquired by F5 Networks

Automatically Start Laravel Horizon with Vessel

Laravel Nova - Trimmed Field Pattern

Exa - A replacement for 'ls'

Laravel Forge - LetsEncrypt Certificates Failing to Renew