Transforming WebSocket Data into Minute-Based HLOC Bars Using NodeJS
11 January 2024
As WebSockets gain momentum in today's tech landscape, it's no surprise that budding developers are eager to dive in and explore this dynamic technology. While setting up a server-client duo and tinkering with WebSocket data can be a thrilling experience, the true magic lies in accessing WebSockets that deliver real-time, actionable data.
The primary objective of this tutorial is to convert the raw data retrieved from WebSockets into minute-based High, Low, Open, and Close (HLOC) bars. This conversion enables developers and analysts to gain insights into market trends over one-minute intervals.
Embark on this tutorial quickly, even if you're new to Node.js! While having some programming experience, especially with JavaScript, could be advantageous, it's not mandatory. If JavaScript and NodeJS aren't your go-to languages, fear not! You can also explore WebSocket implementations using Python and Golang.
Let's outline our objective before we delve into setting up and coding. We're gearing up to fetch real-time Forex data through a WebSocket connection and transform it into minute-based HLOC bars. Our gateway? The TraderMade WebSocket API on the server side. Please refer to the documentation section for more information.
Here's our roadmap, distilled into four simple steps:
1) Acquire NodeJS and configure your environment.
2) Activate a free trial and obtain your API Key.
3) Setting Up a NodeJs WebSocket Client to Receive Data
4) Transforming WebSocket data into Minute-Based HLOC bars using Node.js
Let's Dive In!
Acquire NodeJS and configure your environment
For Windows and Mac Users: Begin by downloading and installing NodeJs.
For Linux Enthusiasts: Use the command apt-get to install NodeJs.
After installing NodeJs, fire your command prompt (Windows) or terminal (Mac/Linux). Let's ensure we have the necessary dependencies. For our client, we're leveraging the "ws" library. Install it using the following command:
npm install ws
Activate a free trial and obtain your API Key:
1) Navigate to the article detailing how to initiate a free 14-day trial.
2) Once we have acquired the API key, store it securely. We will need it shortly!
Setting Up a NodeJs WebSocket Client to Receive Data
1) Locate your NodeJS installation directory. Here, create a new file named forexWsClient.js. Open this file using your preferred code editor: Atom, VS Code, Notepad (Windows) or any text editor in Linux.
2) Now, let's introduce some code magic:
const WebSocket = require ('ws');
const ws = new WebSocket ('wss://marketdata.tradermade.com/feedadv');
First, we'll integrate the WebSocket object, establishing a connection to Tradermade via the secure WSS URL. After successfully connecting, we'll dispatch our API key—obtained during the WebSocket trial sign-up—alongside the specific symbols we're keen on fetching from the server. For instance, we'll target EURUSD.
ws.on('open', function open() {
ws.send("{"userKey":"streaming_api_key", "symbol":"EURUSD"}");
});
ws.on('message', function incoming(data) {
if(data != "Connected"){
data = JSON.parse(data)
console.log(data)
}
});
Running Your WebSocket Client:
1) Save your forexWsClient.js file.
2) Navigate to its location in your terminal or command prompt.
3) Execute the following command to kickstart your client:
node forexWsClient.js
And There You Have It!
Congratulations! You've successfully set up a NodeJs WebSocket client to receive real-time Forex data. Now, watch as the financial world unfolds right before your eyes, thanks to this seamless integration.
{"symbol":"EURUSD","ts":"1704452465709","bid":1.09137,"ask":1.09139,"mid":1.09138}
{"symbol":"EURUSD","ts":"1704452465813","bid":1.09138,"ask":1.09139,"mid":1.091385}
{"symbol":"EURUSD","ts":"1704452466099","bid":1.09137,"ask":1.09139,"mid":1.09138}
{"symbol":"EURUSD","ts":"1704452466099","bid":1.09137,"ask":1.09139,"mid":1.09138}
{"symbol":"EURUSD","ts":"1704452466754","bid":1.09138,"ask":1.09139,"mid":1.091385}
{"symbol":"EURUSD","ts":"1704452467093","bid":1.09138,"ask":1.0914,"mid":1.09139}
{"symbol":"EURUSD","ts":"1704452467093","bid":1.09138,"ask":1.0914,"mid":1.09139}
{"symbol":"EURUSD","ts":"1704452468266","bid":1.09138,"ask":1.09139,"mid":1.091385}
{"symbol":"EURUSD","ts":"1704452468317","bid":1.09138,"ask":1.09138,"mid":1.09138}
{"symbol":"EURUSD","ts":"1704452468338","bid":1.09136,"ask":1.09138,"mid":1.09137}
{"symbol":"EURUSD","ts":"1704452468351","bid":1.09136,"ask":1.09137,"mid":1.091365}
{"symbol":"EURUSD","ts":"1704452468353","bid":1.09135,"ask":1.09137,"mid":1.09136}
{"symbol":"EURUSD","ts":"1704452468414","bid":1.09135,"ask":1.09138,"mid":1.091365}
{"symbol":"EURUSD","ts":"1704452468423","bid":1.09138,"ask":1.09138,"mid":1.09138}
{"symbol":"EURUSD","ts":"1704452468266","bid":1.09138,"ask":1.09139,"mid":1.091385}
{"symbol":"EURUSD","ts":"1704452468317","bid":1.09138,"ask":1.09138,"mid":1.09138}
{"symbol":"EURUSD","ts":"1704452468338","bid":1.09136,"ask":1.09138,"mid":1.09137}
{"symbol":"EURUSD","ts":"1704452468351","bid":1.09136,"ask":1.09137,"mid":1.091365}
{"symbol":"EURUSD","ts":"1704452468353","bid":1.09135,"ask":1.09137,"mid":1.09136}
{"symbol":"EURUSD","ts":"1704452468414","bid":1.09135,"ask":1.09138,"mid":1.091365}
{"symbol":"EURUSD","ts":"1704452468423","bid":1.09138,"ask":1.09138,"mid":1.09138}
{"symbol":"EURUSD","ts":"1704452468494","bid":1.09138,"ask":1.0914,"mid":1.09139}
{"symbol":"EURUSD","ts":"1704452469311","bid":1.09138,"ask":1.09139,"mid":1.091385}
{"symbol":"EURUSD","ts":"1704452469311","bid":1.09138,"ask":1.09139,"mid":1.091385}
{"symbol":"EURUSD","ts":"1704452470123","bid":1.09138,"ask":1.0914,"mid":1.09139}
{"symbol":"EURUSD","ts":"1704452470394","bid":1.09139,"ask":1.0914,"mid":1.091395}
{"symbol":"EURUSD","ts":"1704452470413","bid":1.09139,"ask":1.09141,"mid":1.0914}
{"symbol":"EURUSD","ts":"1704452470123","bid":1.09138,"ask":1.0914,"mid":1.09139}
{"symbol":"EURUSD","ts":"1704452470394","bid":1.09139,"ask":1.0914,"mid":1.091395}
{"symbol":"EURUSD","ts":"1704452470413","bid":1.09139,"ask":1.09141,"mid":1.0914}
{"symbol":"EURUSD","ts":"1704452471383","bid":1.09139,"ask":1.09142,"mid":1.091405}
{"symbol":"EURUSD","ts":"1704452471699","bid":1.0914,"ask":1.09142,"mid":1.09141}
Transforming WebSocket data into Minute-Based HLOC bars using Node.js
Required Modules
The script utilizes two essential Node.js modules:
1) Fs: The File System module allows reading from and writing to files.
2) Luxon: A JavaScript library for working with dates and times.
// Required Modules
const fs = require('fs'); // File System module
const { DateTime } = require('luxon'); // Luxon library for date-time formatting
File Paths
1) inputFilePath: Specifies the path to the input JSON file containing the WebSocket data.
2) outputFilePath: Specifies the path where the processed HLOC data will be saved.
// File Paths const inputFilePath = 'path/to/your/input/file.json'; // Update with your input file path const outputFilePath = 'path/to/your/output/file.json'; // Update with your output file path
Reading Input Data
The script starts by reading the input file asynchronously using fs.readFile. The file content is split by newline characters ( ), and each line is parsed as a JSON object. This approach assumes that each line in the input file represents a distinct WebSocket message in JSON format.
fs.readFile(inputFilePath, 'utf8', (err, data) => {
if (err) {
console.error('Error reading input file:', err);
return;
}
let jsonData;
try {
jsonData = data.split('
').filter(line => line.trim() !== '').map(line => {
try {
return JSON.parse(line);
} catch (error) {
console.error('Error parsing JSON line:', error);
return null;
}
}).filter(Boolean);
} catch (error) {
console.error('Error processing JSON data:', error);
return;
}
if (!jsonData || jsonData.length === 0) {
console.error('No valid JSON data found in the input file.');
return;
}
Grouping Data by Time Intervals
The groupDataByInterval function plays a pivotal role in organizing the data. It groups the WebSocket data based on a predefined time interval (1 minute). Here's how it works:
1) Each data point's timestamp (ts) is converted into a numerical value.
2) This timestamp is then divided by the interval duration (60,000 milliseconds) and rounded down to determine the group (or interval) each data point belongs to.
3) Data points falling within the same interval are grouped.
const groupedData = groupDataByInterval(jsonData, 60000);
const hlocData = calculateHLOC(groupedData);
fs.writeFile(outputFilePath, JSON.stringify(hlocData, null, 2), (err) => {
if (err) {
console.error('Error writing output file:', err);
} else {
console.log('HLOC data written to', outputFilePath);
}
});
});
function groupDataByInterval(data, interval) {
const groupedData = {};
data.forEach(item => {
const intervalKey = Math.floor(item.ts / interval) * interval;
if (!groupedData[intervalKey]) {
groupedData[intervalKey] = [];
}
groupedData[intervalKey].push(item);
});
return groupedData;
}
Calculating HLOC Data
Once intervals group the data, the calculateHLOC function computes the essential HLOC values for each interval:
1) Open: The mid-value of the first data point within the interval.
2) High: The maximum mid-value observed within the interval.
3) Low: The minimum mid-value observed within the interval.
4) Close: The mid-value of the last data point within the interval.
function calculateHLOC(groupedData) {
const hlocData = [];
for (const intervalKey in groupedData) {
const intervalData = groupedData[intervalKey];
const open = roundToFiveDecimalPlaces(intervalData[0].mid);
const close = roundToFiveDecimalPlaces(intervalData[intervalData.length - 1].mid);
const high = roundToFiveDecimalPlaces(Math.max(...intervalData.map(item => item.mid)));
const low = roundToFiveDecimalPlaces(Math.min(...intervalData.map(item => item.mid)));
hlocData.push({
interval: Number(intervalKey),
intervalDateTime: formatDateTime(Number(intervalKey)),
open,
high,
low,
close
});
}
return hlocData;
}
Formatting and Writing Output
After calculating the HLOC values for each interval, the data is formatted into a structured JSON format containing:
1) Interval: A timestamp representing the start of the minute interval.
2) IntervalDateTime: The formatted date-time corresponding to the interval timestamp.
3) Open, High, Low, Close: The computed HLOC values.
Finally, the processed HLOC data is written to an output JSON file using fs.writeFile.
function formatDateTime(timestamp) {
return DateTime.fromMillis(timestamp).toUTC().toFormat('yyyy-MM-dd HH:mm:ss');
}
function roundToFiveDecimalPlaces(value) {
return Math.round(value * 1e5) / 1e5;
}
} else {
console.log('HLOC data written to', outputFilePath);
}
});
Hurray! We have our Output ready
{
"interval": 1704453720000,
"intervalDateTime": "2024-01-05 11:22:00",
"open": 1.09152,
"high": 1.09152,
"low": 1.09132,
"close": 1.09132
},
{
"interval": 1704453780000,
"intervalDateTime": "2024-01-05 11:23:00",
"open": 1.09133,
"high": 1.09153,
"low": 1.09132,
"close": 1.09152
},
{
"interval": 1704453840000,
"intervalDateTime": "2024-01-05 11:24:00",
"open": 1.09153,
"high": 1.09153,
"low": 1.09141,
"close": 1.09142
}
Conclusion
This tutorial offers a streamlined approach to transforming WebSocket data into actionable insights. By converting raw data into minute-based HLOC bars, developers and analysts can evaluate market trends, identify patterns, and make informed decisions. The script's modular design, leveraging the Luxon library for date-time operations, ensures efficiency and accuracy in data processing and analysis.
