𝕽𝖚𝖆𝖎𝖉𝖍𝖗𝖎𝖌𝖍

       🅸 🅰🅼 🆃🅷🅴 🅻🅰🆆. 
 𝕽𝖚𝖆𝖎𝖉𝖍𝖗𝖎𝖌𝖍 𝖋𝖊𝖆𝖙𝖍𝖊𝖗𝖘𝖙𝖔𝖓𝖊𝖍𝖆𝖚𝖌𝖍 
  • 3 Posts
  • 388 Comments
Joined 3 years ago
cake
Cake day: August 26th, 2022

help-circle





  • 2 opponents have acquisition, one looks like they’re firing, there’s a third ready for a snap shot, and they all have AKs.

    Player is… reloading a handgun.

    I know it’s a video game, but my suspension of disbelief has limits. This isn’t sci-fi; there are no magic shields or SuperArmor; so player only survives if

    • the opponents have Storm Trooper levels of aim capability
    • Player is wearing level… hell, I don’t even know what body armor is effective against 7.62x39 at - what - no more than 7 meters here. And it’s going to hurt like hell in any case
    • the opponents are all out of ammo

    I mean, utter, blinding incompetence aside, player 1 is dead. Even with a massive plot shield, it would be hard to swallow.

    Maybe they stole some armor cloth from the John Wick universe; that stuff is sci-fi level good.





  • In the Verge article, are you talking about the table the the “presumably” qualifier in the table column headers? If so, not only is it clear they don’t know what, exactly, is a attributable to the costs, but also that they mention “gross pay”, which is AKA “compensation.” When a company refers to compensation, they include all benefits: 401k contributions, the value of health insurance, vacation time, social security, bonuses, and any other benefits. When I was running development organizations, a developer who cost me $180k was probably only taking $90k of that home. The rest of it went to benefits. The rule of thumb was for every dollar of salary negotiated, I had to budget 1.5-2x that amount. The numbers in “Presumably: Gross pay” column are very likely cost-to-company, not take-home pay.

    I have some serious questions about the data from “h1bdata.info”. It claims one software engineer has a salary of $25,304,885? They’ve got some pretty outlandish salaries in there; a program manager in NY making $2,400,000? I’m sceptical about the source of the data on that website. The vast number of the salaries for engineers, even in that table, are in the range of $100k - 180k, largely dependent on location, and a far cry from a take-home salary of 500,000€.


  • Nobody is paying software developers 500.000€. It might cost the company that much, but no developers are making that much. The highest software engineer salaries are still in the US, and the average is $120k. High-end salaries are $160k; you might creep up a little more than that, but that’s also location specific. Silicon Valley salaries might be higher, but then, it costs far more to live in that area.

    In any case, the question is ROI. If you have to spend $500,000 to address some sites that are being clever about wasting your scrapers’ time, is that data worth it? Are you going to make your $500k back? And you have to keep spending it, because people keep changing tactics and putting in new mechanisms to ruin your business model. Really, the only time this sort of investment makes sense is when you’re breaking into a bank and are going to get a big pay-out in ransomware or outright theft. Getting the contents of my blog is never going to be worth the investment.

    Your assumption is that slowly served content is considered not worth scraping. If that’s the case, then it’s easy enough for people to prevent their content from being scraped: put in sufficient delays. This is an actual a method for addressing spam: add a delay in each interaction. Even relatively small delays add up and cost spammers money, especially if you run a large email service and do it at scale.

    Make the web a little slower. Add a few seconds to each request, on every web site. Humans might notice, but probably not enough to be a big bother, but the impact on data harvesters will be huge.

    If you think this isn’t the defense, consider how almost every Cloudflare interaction - and an increasingly large number of other sites - are including time-wasting front pages. They usually say something like “making sure you’re human” with a spinning disk, but really all they need to be doing is adding 10 seconds to each request. If a scraper of trying to indeed only a million pages a day, and each page adds a 10s delay, that’s wasting 2,700 hours of scraper computer time. And they’re trying to scrape far more than a million pages a day; it’s estimated (they don’t reveal the actual number) that Google indexes billions of pages every day.

    This is good, though; I’m going to go change the rate limit on my web server; maybe those genius software developers will set a timeout such that they move on before they get any content from my site.



  • Ah, that’s where tuning comes in. Look at the logs, take the average time-out, and tune the tarpit to return a minimum payload consisting of a minimal HTML containing a single, slightly different URL back to the tar pit. Or, better yet, JavaScript that loads a single page of tarpit URLs very slowly. Bots have to be able to run JS, or else they’re missing half the content on the web. I’m sure someone has created a JS forkbomb.

    Variety is the spice of life. AI botnet blacklists are probably the better solution for web content; you can run ssh on a different port and run a tarpit on the standard port, and it will barely affect you. But for the web, if you’re running a web server you probably want visitors, and tarpits would be harder to set up to catch only bots.





  • A good tar pit will reduce your bandwidth. Tarpits aren’t about shoving useless data at bots; they’re about responding as slow as possible to keep the bot connected for as long as possible while giving it nothing.

    Endlessh accepts the connection and then… does nothing. It doesn’t even actually perform the SSL negotiation. It just very… slowly… sends… an endless preamble, until the bot gives up.

    As I write, my Internet-facing SSH tarpit currently has 27 clients trapped in it. A few of these have been connected for weeks. In one particular spike it had 1,378 clients trapped at once, lasting about 20 hours.