Lab 6 — DPS909

Interacting with Telescope

This week I had the chance to interact with the Telescope project. The installation process and setting up the environment went pretty well. This is due to the experience I had in the previous month installing the Mattermost server.

The task for this week was to pull the posts from Telescope API and check the links inside of these posts by using our URL link checker. I decided to create a function that sends a request to the local server which returns a JSON object with posts’ ids and URLs. Then I extracted all the posts ids and store them in the array. Then I iterate through the array of ids and called another function called writePostToFile with the id parameter.

const writePostToFile = async (post,id) => {
fetch(post)
.then(
res =>
new Promise((resolve, reject) => {
const path = `./telescope/posts/${id}`;
const dest = fs.createWriteStream(path);
res.body.pipe(dest);
res.body.on("end", () => resolve(path));
dest.on("error", reject);
})
)
.then(path => {
console.log(path);
let p = `${path}`;
fileHandler.processFile(p);
});

This function sends a request to the local server with the specified post id, retrieves the text of the post and creates a new file with the post’s content. When the file is created it passes the file to the processFile function.

const processFile =  (filename,json) => {
// Create stream with the file
let s = fs.createReadStream(filename);
return new Promise((resolve, reject) => {
s.on("data", (buf) => {
// Get all the URL links from the file
urlList = buf
.toString()
.match(/(http|https)(:\/\/)([\w+\-&@`~#$%^*.=/?:]+)/gi);
// console.log(`${filename} ${urlList}`)
});

s.on("end", async () => {
var responseStatusByUrl = [];
var statusResponseForUrl;
//Iterate through the links and check their status
await Promise.all(
urlList.map(async (url) => {
try {
const urlTest = await fetch(url, { method: "head", timeout: 1500 });
statusResponseForUrl = { url: `${url}`, status: `${urlTest.status}` };
responseStatusByUrl.push(statusResponseForUrl);
} catch (error) {
statusResponseForUrl = { url: `${url}`, status: "UNKNOWN" };
responseStatusByUrl.push(statusResponseForUrl);
}
})
);
if (json) {
console.log(JSON.stringify(responseStatusByUrl));
} else {
printResponse(responseStatusByUrl);
}
process.exit(checkStatus(responseStatusByUrl));
});
s.on("error",error => reject(error));
});
}

This function receives a file, extracts all the URLs, and checks all the URLs statuses.

The only issue I had while writing the code is the use of Promises which I am sure can be improved in my code. Hope to fix it in the future.

Commit of this work — -> Lab 6.