0

Looking for some assistance or ideas on how to grab the latest version of a Tomcat release from their website or other website.

I came across these Stack Overflow/Stack Exchange links that were relevant below:

..But unfortunately both led to some dead-ends. It did give me some ideas though.

I have some installation scripts where the download URL is predictable based on the version number of the software, so if there's a consistent way of scraping for the latest version of the software I simply add a variable into the script for it to curl / grep for that version on the text and use that variable for download links, untaring, moving, etc.

Hoping to do something with Tomcat in the same vein. A installation script I won't need to keep updating (unless they change their site).

Any sort of thoughts, ideas, are appreciated.

I've also looked at http://tomcat.apache.org/whichversion.html which lists the "Latest Released Version" and this seems like maybe the best, more reliable location to grab the version, but I had trouble getting that with Curl / Grep because of the table structure. I am only just scratching the surface with learning those commands, so maybe someone more well versed could get that working, otherwise I'm definitely open to other thoughts!

curl -i https://api.github.com/repos/apache/tomcat/tags | grep '"name"' | head -1 | egrep -o "([0-9]{1,}\.)+[0-9]{1,}"

It seems to work but I am not sure if I am entirely happy with it. An additional use-case that I thought of, is it would be nice to grab the latest version of each major version instead of just the absolute latest.

Do you have suggestions on how you would parse the highest number with the link you provided, and/or how to start in one of those directories and parse the highest there? That way for example I could create a latest version install script for Tomcat 7, 8, 9, etc.

Thank you!! (Like I said I am just getting started, so please my apologies with the lack of knowledge but I do appreciate digging in and understanding how the commands work or reached their conclusion)

8
  • 1
    You should not upgrade tomcat without prior reading of the changelog, even minor security fix may break your application. – Tensibai Mar 21 '19 at 15:01
  • Hi @Tensibai thanks for the advise. The goal here is to create a version agnostic installation script that will always grab the latest version. So that when we do approve the latest version, I can simply run the script that will already be present on the machines or systems. Assume here that we already taking the necessary precautions surrounding the version, and just care to have a script to grab the latest for a version (Tomcat 7, Tomcat 8, Tomcat 9, etc). Appreciate your concern and insight! – Keegan Jacobson Mar 21 '19 at 15:06
  • Best course of action would be to pass as a parameter the version to install, specially as there's no "latest" tomcat, but 3 latest depending on major version, so taking the latest by date may be 7, 8 or 9 and you'll have to at least pass this major as parameter, better pass the whole desired version directly. – Tensibai Mar 21 '19 at 15:44
  • What I'm actually going to do is have three different installation scripts, one for 7, 8, and 9 (and then in the future one for each of the future versions) but I want to be able to grab the latest release for a particular major version. – Keegan Jacobson Mar 21 '19 at 20:19
  • And you'll have 3 scripts to maintain after that, that's just a bad idea. – Tensibai Mar 22 '19 at 8:45
0

I've created a couple crude methods of grabbing the latest major versions of releases, as well as absolute latest.

# Grab Latest Tomcat 7
curl -i https://www-us.apache.org/dist/tomcat/tomcat-7/ | grep -Po '(?<=(<a href="v)).*(?=/">v)'

# Grab Latest Tomcat 8
curl -i https://www-us.apache.org/dist/tomcat/tomcat-8/ | grep -Po '(?<=(<a href="v)).*(?=/">v)'

# Grab Latest Tomcat 9
curl -i https://www-us.apache.org/dist/tomcat/tomcat-9/ | grep -Po '(?<=(<a href="v)).*(?=/">v)'

# Grab Latest Tomcat
curl -i https://api.github.com/repos/apache/tomcat/tags | grep '"name"' | head -1 | egrep -o "([0-9]{1,}\.)+[0-9]{1,}"

It's a bit of a hack where I'm looking for text in-between two values on a web page to grab the version number. It's crude and I'm sure there's a more elegant way to grab this, but this what I was able to figure out with my skillset! Maybe it can help someone in the future, but if anyone has better ideas I'm all ears.

0

In general, I would not recommend scraping version numbers and download URLs from websites. You will end up having to modify your scripts when they inevitably end up breaking — I have learned this the hard way time and time again.

For example, take the following script I wrote in 2016 to scrape the latest Tomcat version:

curl https://tomcat.apache.org/whichversion.html | awk -n '
/class="detail-table"/ {
    phase = 1
}
phase == 1 && /<th>/ {
    target_column++
    if (/Actual release revision/) phase = 2
}
phase == 2 && /<tr>/ {
    current_column = 0
}
phase == 2 && /<td>/ {
    current_column++
}
phase == 2 && current_column == target_column && !/alpha|beta|superceded/ {
    print substr($0, match($0, /[0-9]+\.[0-9]+\.[0-9]+/), RLENGTH)
    exit
}'

Unfortunately, that script no longer works, and I do not know when it broke. No amount of fancy scripting will save you from unexpected structural changes to a website you are scraping.

Sure, maybe a little bit of finagling will fix this one particular script, but if you try to download all your dependencies this way, you'll be playing catch-up till the end of time. Aim for a solution that you only need to code once.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.