Up(sun) and running with infrastructure processes on Blackfire

Encountering an event that impacts application behavior and performance is inevitable, however, implementing an Observability strategy for production websites which benefits from as much context as possible gives you the best overview of your applications. Helping you to identify potential bottlenecks and ensure a seamless user experience.

Adding infrastructure information to an Observability dashboard lets you understand what was done on the infrastructure side and, if necessary, diagnose problems as quickly as possible. Is this resource consumption spike due to a traffic spike or due to something that occurred on the infrastructure side? Is a request delay due to a newly launched ad or a newly deployed service? Each result requires a different response, so any added context to your Observability dashboard helps you to split apart internal and external causes.

In the meantime, Upsun gives you access to all of the infrastructure processes that occur in your environment/project/integration/maintenance and lets you trigger an action for each of them. For each infrastructure process, you can decide what to do: send a notification to a specific Slack channel, send information to a custom endpoint (webhook), execute a dedicated script, send information to Newrelic, the list goes on. 

An Activity script is a piece of code, written in ES2021 (Javascript) that can be executed to do anything you want. The glue between those two, Activity scripts and Infrastructure processes, is done using an integration.

This integration needs to be configured with parameters, defining which action you want to execute, on which environments, and which events and corresponding states trigger this Activity script. 

• Action: For the purpose of this article, we will use the action script (=execute an Activity script) but you can find the full list of possible actions here. We will also need to define which script (option --file) we want to execute.
• Environment: It could be set to all environments, but in our case, let's say we just want to handle infrastructure processes from the production environment (main).
• Event: In our case, we want to send to my monitoring dashboard any infrastructure event, but if you want to fine tune it, a complete list of possible infrastructure events can be founded here (= Activity script types).
• State: There are 3 available states, pending, in_progress, and complete. In our case, pending operations won’t have any impact on your application (Upsun internal usage) so we will focus on in_progress and complete states.
   

Upsun uses Blackfire as its integrated APM/Continuous profiling tool, and its usage is included (as far as I’m writing this article) on all of your Upsun projects. 

Blackfire provides a REST endpoint to add markers, using a simple curl call with your credentials (blackfire_server_id and blackfire_server_token) and the message you want to display on your Blackfire Timeline.

We now have great ingredients for our recipe: how to send infrastructure processes to your Blackfire monitoring. Let’s add Upsun infrastructure processes in our Blackfire timeline. 

Spot Upsun infrastructure processes in your Blackfire timeline

To spot Upsun infrastructure processes to the Blackfire timeline, we will need:

• An Activity script, which will add new markers on your Blackfire environment.
• An Activity script integration to trigger this Activity script on any occurring infrastructure processes.

These are the minimum steps for your Upsun environment to send infrastructure processes to your Blackfire environment. 

Go to the root of your local project source code and follow these steps: 

1. Create a new Javascript file (our Activity script file) at the root of your source code,  blackfire-notifier.js (source here) with the following command line:

curl -L https://raw.githubusercontent.com/upsun/snippets/main/src/blackfire-notifier.js > blackfire-notifier.js
git add blackfire-notifier.js && git commit -m "Add blackfire-notifier.js"

This blackfire-notifier.js Javascript file is using 2 environment variables, blackfire_server_id and blackfire_server_token to send corresponding infrastructure processe info (state and type, ex: (start|stop) Florent Huck redeployed environment Main ) to your Blackfire environment, using the dedicated Blackfire REST endpoint to add markers.

2. Add an Activity script integration for this new blackfire-notifier.js script:

upsun integration:add --type=script --file ./blackfire-notifier.js --events \* --states in_progress,complete --environments main

With this integration, Upsun will execute blackfire-notifier.js script for all infrastructure processes (with state in_progress or complete) on your main environment.

3. Forward Blackfire credentials, BLACKFIRE_SERVER_ID and BLACKFIRE_SERVER_TOKEN to your activity script:

From the root of your Upsun project, execute the 2 following command lines:

upsun project:curl /integrations/<INTEGRATION_ID>/variables -X POST -d '{"name": "blackfire_server_id", "value": "'$(upsun ssh 'echo $BLACKFIRE_SERVER_ID')'", "is_sensitive": true, "is_json": false}'
upsun project:curl /integrations/<INTEGRATION_ID>/variables -X POST -d '{"name": "blackfire_server_token", "value": "'$(upsun ssh 'echo $BLACKFIRE_SERVER_TOKEN')'", "is_sensitive": true, "is_json": false}'

4. Test it. 

To test the integration of your Activity script, just trigger an infrastructure process on your project:

upsun project:clear-build-cache && upsun environment:redeploy -y

New markers should appear on the Blackfire side to spot the start and the end of the process:

5. Debug it and update Activity script.

If you want to get access to the latest Activity Script logs, you can use the following command: 

upsun integration:activity:log <INTEGRATION_ID>

As the content of the blackfire-notifier.js script has been saved in cache when creating the integration, you need to update the integration each time an update has been done on this script, by using the following command:

upsun integration:update --file ./blackfire-notifier.js <INTEGRATION_ID>

Et voilà, as soon as an infrastructure process occurs on your main environment, this Activity script will add more context to your Blackfire timeline by adding new markers. You will gain a holistic view of your application’s performance and enhance your ability to diagnose and resolve problems swiftly. 

By leveraging the full potential of Blackfire and incorporating Upsun infrastructure processes, you are better equipped to manage and optimize your application’s performance. Embrace this integrated approach to stay ahead of potential issues, improve operational efficiency, and provide a reliable service to your users.

Stay up to date on the latest from us over on our social media and community channels: Dev.to, Reddit, and Discord. Happy monitoring.