Has anyone created a greasmonkey/tampermonkey script or used any sort of site scraper to extract their current flavor inventory from alltheflavors.com? The JSON export does not include on-hand inventory levels and the CSV export is (still) broken. My initial need for this data was to build a shopping list of the things I'm low on without having to inspect hundreds of 10-15ml bottles. I opened a support ticket on the site (as I bought a year sub about 10 months ago) and it has now been 10 full days without anything beyond the initial automated reply - needless to say I'm not renewing & will be moving to an offline calculator/inventory management solution, but I would really love to be able to somehow extract the on-hand levels without having to click into every single flavor.
Thanks!
- Go to https://alltheflavors.com/users/your_username/flavors
- Ensure you are logged in
- Then run this in the JS console
$('.togglemore').click();
$('tr.stashed').map((i, v) => {
const $el = $(v);
return {
name: $el.find('td:nth-child(2)').text().trim(),
stock: parseFloat($el.next().find('input[name="user_flavor[stock_level]"]').val())
};
}).get()
);
It will output a JSON array of your flavor names with the corresponding stock levels. You'll need to extend it if you want more data.
EDIT: Added a second line of JS that will expand all the toggle first.
Damn, you have no idea how much more complex I was thinking about the Phyton script, lol. Thank god I'm no dev for a living or I wouldn't have cash for cotton
Luckily all the HTML is on one page if you use the /users/x/flavors route, so no follow-up requests needed. If I were doing anything else for a living, I'd be broke too :)
I did go ahead and extend it to include the cost and cost per ml. Also - worth mentioning, the /users/username/flavors route ONLY works if you have made your stash public. The other path, via the dropdown menu "my flavors", will work without a publicly visible stash, but is paginated and you'll have to run it once per page (easier to just make stash public, run, then if desired toggle public visibility back off.)
JSON.stringify( $('tr.stashed').map((i, v) => { const $el = $(v); return { name: $el.find('td:nth-child(2)').text().trim(), stock: parseFloat($el.next().find('input[name="user_flavor[stock_level]"]').val()),purch_vol: parseFloat($el.next().find('input[name="user_flavor[volume]"]').val()),cost: parseFloat($el.next().find('input[name="user_flavor[cost]"]').val()), cost_per_ml: parseFloat($el.next().find('input[name="user_flavor[cost_per_ml]"]').val()) }; }).get() );
Right, and if someone has the time or inclination they can use something like puppeteer to control a headless browser and do all the paginated requests and then aggregate the results, but that seems like overkill if you're just dumping this once.
This looks very close, but returns nulls for every row I haven't pre-expanded (by clicking the "greater than" symbol to the right of the row), and does need to be run on each page of the flavor stash (not a big deal at all - I only have like 3 pages) Any thoughts re: pre-expanding the rows?
Sure, just run this before the JSON.stringify:
$('.togglemore').click();
It will take several seconds to expand all the flavors.
Beautiful! Much appreciated!
(and just in case it wasn't clear to others following along - this combo works brilliantly and dumps exactly the data I need, in an easily used format. If you don't/can't work w/ the json directly, there are online converters which will let you paste the resulting array in and return it in csv or excel format)
It's kind of easy to do a Python script for web scraping, but I really have no idea what data do you actually need (i'm not familiar with this website).
You want the data from all items here : https://www.alltheflavors.com/flavors#
And each item should get the data from here: https://www.alltheflavors.com/flavors/capella-27-bears
Is that it?
I've never tried dynamic websites with search functions like this, but I can take a look on the weekend if no one else do it before me.
I'm currently learning python, but I'm definitely not at a level where I could do that myself yet, sadly
It's really not that difficult dude. Python is very powerful and there's a ton of tutorials and premade stuff, and a lot of libraries as well. 99% of the time I'm just adapting old or other people's code. Bear in mind I'm no professional dev, but I do a few scripts to automate some tasks or use APIs. As for the scraper, you can literally search "python scraper" on google and you'll see a lot of examples. Basic programming knowledge can get you farther than you expect these days, haha.
No - I'm really not interested in the public flavor database for this use case, I'm after my personal user data related to which flavors I've recorded as being "on-hand" on the site. The reply from /u/tesladelmar is closer to my goal but not quite there. (but thanks for the effort!)
Oh, I've seen now, np, he solved in a much less complicated way that I would do, lol.
Yep - very slick little approach! It's not something I'd want to do regularly, nor is it particularly user-friendly, but since this is a one-off, "get my data off the site" situation, it's really the perfect solution. I can't imagine it would be all that difficult to extend as a userscript if someone were so inclined.
I have $5 on it if anyone can do it successfully. Dead serious.
Looks like you owe /u/tesladelmar a fiver!
Did they actually do it?????? I'm serious. If I was a rich man I would just buy the fucking site. You guys have no idea how much money that site saved me over the years.
Yep. Top comment is a dead simple command you can paste into the js console and get a dump of your flavor inventory. I grabbed my inventory dump, and promptly stopped my recurring PayPal payment. I've loved the functionality of atf this past year but I'm not willing to continue paying for it with the outages and total lack of response to support requests. I'll still use the free functionality for discovering new recipes but anything beyond that I'm taking offline.
in for answer
The response given by /u/tesladelmar works a treat. I haven't parsed out the json array for import into my offline calc yet, but it definitely produces the necessary output. I'll have to wait until I get home to actually complete the export/import but I can positively say that it does exactly what I asked for.