Some bad sites gives in location header URL without proper encoding #189

marcocarvalho · Sep 30, 2012

require 'open-uri'

utl = "http://www.grupogonzaga.com.br/tabid/2076/ref/-Resid%C3%AAncia-Curitiba-Tatuquara-1-quarto/fic/875756.aspx" 
o = open(url)

#OpenURI::HTTPError: 301 Moved Permanently (Invalid Location URI)
#   from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:342:in `rescue in open_http'
#   from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:339:in `open_http'
#   from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:775:in `buffer_open'
#   from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:203:in `block in open_loop'
#   from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:201:in `catch'
#   from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:201:in `open_loop'
#   from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:146:in `open_uri'
#   from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:677:in `open'
#   from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:33:in `open'
#   from (irb):68
#   from /usr/local/rvm/rubies/ruby-1.9.3-p194/bin/irb:16:in `<main>'

becouse this url returns a redirect with improper encoding:

resp = Net::HTTP.get_response(URI.encode(url))
resp.header['location']
# "http://www.grupogonzaga.com.br/Alugar/Ficha-do-im\xC3\xB3vel/fic/875756.aspx?ref=-Resid\xC3\xAAncia-Curitiba-Tatuquara-1-quarto"

Note the \xC3\xB3 bad encoded location return.

Added URI.encode to header['location'] on open-uri module.

zzak · Nov 19, 2012

Closing this, please refer to Feature #7396 in redmine

marcocarvalho Some bad sites gives in location header URL without proper encoding f…
…ixed with URI.encode

5a9c5c0

zzak closed this Nov 19, 2012

toddmazierski added a commit to lolibrarian/nypl-tweetwall that referenced this pull request Jul 25, 2016

toddmazierski Make URL redirect following more reliable
Use Curb to work around ruby/ruby#189
5bbe5c0

ruby/ruby

Some bad sites gives in location header URL without proper encoding #189

Reviewers

Assignees

Labels

Projects

Milestone

2 participants