Packaging and Disribution(2) • gemcutter.org: • https://github.com/rubygems/gemcutter • You can use `gem yank` command after you did invoke `gem i gemcutter`. • rubygems.org: • gemcutter.org was renamed to rubygems.org. • bundler: • 2010: Released to 1.0.0 version.
A History of Ruby(Short Ver.) • Ruby 1.8.7 bundled a lot of library named standard library a.k.a stdlib. 2008 • Ruby 1.9.x bundled rubygems that is ruby library packaging and distributing. 2009 • Many of libraries was born after that. Example for rspec, nokogiri, rake and thor. It’s start time for Ruby language and library ecosystems. • First Authors of famous library is burnout. 2013 • Some of useful gems are changed maintainers of second generation. • Ruby 2.4, 2.5 2017
Classification of standard libraries This matrix shows number of standard libraries and their classifications in Ruby 2.4. Standard Default Bundled Libraries Gems Gems Pure Ruby 69 1 7 C extensions 23 5 0
What differences these libraries? • Standard Libraries • Upstream: Only Ruby core repository(svn.ruby-lang.org) • Release cycle: 1 year • Default Gems • Upstream: Ruby core repository and GitHub • Release cycle: 1year or maintainers matter • Bundled Gems • Upstream: Only GitHub • Release cycle: Maintainers matter
What number of these libraries - 2.4.0 to 2.5.0 In Ruby 2.4 In Ruby 2.5 Standard Libraries Standard Libraries • Pure ruby: 69 • Pure ruby: 64 (-5) • Extensions: 23 • Extensions: 14 (-9) Default gems Default gems • Pure ruby: 1 • Pure ruby: 6 (+5) • Extensions: 5 • Extensions: 14 (+9) Bundled Gems Bundled Gems • Pure ruby: 7 • Pure ruby: 7 • Extensions: 0 • Extensions: 0
2. Default gems
Inside default gems • `tool/rbinstall.rb` put gemspec files for default gems on Ruby core repository. • We can release default gems to the rubygems.org. It’s a Standard library that seems to be installed as a gem. • Rubygems have a detection method for default gems. >> Gem.loaded_specs["did_you_mean"].default_gem? => false >> require 'openssl' => true >> Gem.loaded_specs["openssl"].default_gem? => true
Status of OpenSSL binding • OpenSSL is already extracted default gems. You can update it separated ruby core releases same as rubygems, rdoc, bigdecimal. • https://github.com/ruby/openssl • It’s maintained by @rhenium • Upstream was changed github repository from svn.ruby-lang.org • He aggressively maintains new feature of openssl
Security release We hard to fix and release these security issue. so all of release maintainer are volunteer work. But If your Ruby supports Default gems/Bundled gems, You can upgrade these gems without upgrading ruby interpreter.
Whats psych? A libyaml wrapper for Ruby It supports bundled libyaml-* sources. You can build psych without libyaml-devel(dev) packages. # Load YAML in to a Ruby object Psych.load('--- foo') # => 'foo' # Emit YAML from a Ruby object Psych.dump("foo") # => "--- foo\n...\n"
JRuby integration • We need to support Java implementation now. • I guarantee compatibility for other implementation used by Travis. In current status, It compatibility was breaking. Because It’s beta status.
Whats rdoc? RDoc produces HTML and online documentation for Ruby projects. # puts listings.product + "\n" + listings.price # # _Generates_ # # Chicken Fried Steak # A well messages pattie, breaded and fried. # # Chicken Fried Steak -- 9.95 # A well messages pattie, breaded and fried. # def initialize(str, safe_level=nil, trim_mode=nil, eoutvar='_erbout') @safe_level = safe_level compiler = make_compiler(trim_mode) set_eoutvar(compiler, eoutvar) @src, @encoding, @frozen_string = *compiler.compile(str) @filename = nil @lineno = 0 end
Current status of rdoc • I moved canonical repository to ruby/rdoc. • I handle a project of rdoc and management releases(not product management) • Our official tracker named “bugs.ruby-lang.org” is high threshold. But GitHub resolve this by pull requests. VS
3. Bundled gems
Inside Bundled gems • It was installed by `tool/rbinstall.rb` on Ruby core repository. • You can see list of Bundled gems via `gems/bundled_gems` • We bundled *.gem and unpacked files to tarball package for Bundled gems. • `make install` installed Bundled gem your box.
Why separated the test framework? The following libraries uses minitest directly in Ruby 2.3: • rubygems • rdoc Other libraries uses test-unit. rubygems and rdoc are developed at github.com/rubygems/rubygems and github.com/ruby/rdoc. We need to support these libraries and their tests.
A problem of test-unit/minitest • It has difference behaviors and namespace from released gems that are test-unit and minitest. • [Bug #9660] test/unit, minitest & bundler • We gave up to maintain compatibility with released gems. • But @zenspider said “I AM against shipping ruby without a testing library.”
Make in Ruby Rake is a Make-like program implemented in Ruby. Tasks and dependencies are specified in standard Ruby syntax. task :awesome do puts :bar end task beat: [:awesome] do puts :buzz end task default: :beat
ruby/rake • Rake was originally created by Jim Weirich, who unfortunately passed away in February 2014. • This repository was originally hosted at github.com/ jimweirich/rake, It has been moved to github.com/ruby/rake by @drbrain • Rake adopted SemVer policy. (v0.9.x to v10.x.y) • @drbrain and @hsbt maintain ruby/rake now.
Concerns of bundled gems • We couldn’t support to compile C extension on Bundled gems. • Bundled gems couldn’t support cross compilation. Ex. We need to make Date gem to bundled gem before extracted from ruby core.
What’s problem of Gamification? • It issue size is too big for human. • I resolved it used default gems and bundled gems. • We can promote standard libraries to default gems, after that we promote it to bundled gems. • I am moving toward the final goal for Gemification.
Pros of Gemification • Maintainers can release gem for bugfix, new feature independent with Ruby core. • Easily backport stable version from develop version. Ruby users can use new feature on stable version. • If upstream is available on GitHub, Ruby users easily send patch via Pull request. • Ruby interpreter developper can concentrate development Ruby internal.
Cons of Gemification • Abandoned and complex dependency on rubygems and bundler. • Maintainers need to maintain ruby core and GitHub repositories both. • It’s hard to maintain compatibility.
How to merge upstream from others I merged upstream into ruby/ruby periodically using following instructions. $ git clone https://github.com/ruby/ruby $ git clone https://github.com/rubygems/rubygems $ cd ruby $ rm -rf lib/rubygems test/rubygtems lib/rubygems.rb $ cp -rf ../../rubygems/rubygems/lib/rubygems ./lib $ cp -rf ../../rubygems/rubygems/lib/rubygems.rb ./lib $ cp -rf ../../rubygems/rubygems/test/rubygems ./test $ git checkout lib/rubygems/LICENSE.txt ruby and rubygems guarantee to work to test and code each other. it’s the same situation for ruby and rdoc
backport is hard rubygems still supports Ruby 1.8. % g show a34fb569e41cd87866e644d92a9df4be89b3cad2 test/rubygems/test_gem_package.rb commit a34fb569e41cd87866e644d92a9df4be89b3cad2 Author: Eric Hodel <email@example.com> (snip) --- test/rubygems/test_gem_package.rb +++ test/rubygems/test_gem_package.rb @@ -638,7 +638,7 @@ class TestGemPackage < Gem::Package::TarTestCase e.message io end - tf.close! + tf.close! if tf.respond_to? :close! end def test_verify_empty
5. RubyGems and Gemification for Ruby 2.5
What’s rubygems RubyGems is a package management framework for Ruby. • rubygems/rubygems.org: • The Ruby community's gem host. • rubygems.org is maintain by infrastructure team of rubygems. It is different team from rubygems cli team. • rubygems/rubygems: • Command line tool of rubygems • Rubygems are created by Seattle.rb
Current problem of rubygems Bundler team funded RubyTogether take over rubygems/ rubygems In the past, We easily discuss maintenance plan and policy about ruby and rubygems to @tenderlove and @drbrain. I hard to backport a patch to rubygems from ruby core. Because these patch only need ruby trunk. It is needless stable versions.
Reserved words on rubygems.org Rubygems block reserved name same as standard libraries. • But you can see: https://rubygems.org/gems/fileutils/versions/ 0.7 • If you install fileutils-0.7, fileutils-0.7 break your ruby environment. Why???
fiddle fiddle is standard library for wrapper of libffi. But fiddle was already reserved another implementation(https:// github.com/bsm/fiddle) on rubygems.org I did coordinate to transfer above namespace and override CRuby implementation(https://github.com/ruby/ fiddle) now.
What we will do? I must offer to transfer ownership of these gems to rubygems.org and overwrite it used by ruby stdlib. My plan for reserved gems: ・Transfer request to owners of reserved gems. ・Removed gemification gems from blacklist on rubygems.org. ・Override reserved gems by standard libraries.
Reducing Ruby package size •In Ruby 2.5, We added “bundler” to default gems. •Bundler will be integrated RubyGems until Ruby 3.0 releasing. •I will promote all of standard libraries to default gems. •I will promote all of default gems without Rubygems dependencies to bundled gems.
Bundle Bundler to Ruby core • Bundler uses rspec. • I made `test-bundler` task that is invoking rspec examples of Bundler. ~/D/r/trunk > mk test-bundler ./miniruby -I../../github.com/ruby/ruby/lib -I. -I.ext/common ../../github.com/ruby/ruby/tool/runruby.rb --extout=.ext -- --disable-gems -C "../../github.com/ruby/ruby" bin/gem install --no-ri --no-rdoc \
Migration status of Rubygems and Bundler • rubygems-2.7.0 uses bundler on only test code. • After bundler-2.0 released, We will target rubygems-3.0 • Ruby 2.5.0 have rubygems(-2.6.13) and bundler(-1.15.4), It helps migration plan of rubygems/bundler.
Activated problem for Rubygems • Rubygems have a activated problem with default gems. • https://bugs.ruby-lang.org/issues/13847 $ ruby -ropenssl -e 'p OpenSSL::VERSION' "2.0.5" $ cat Gemfile # frozen_string_literal: true source "https://rubygems.org" gem 'openssl', '2.0.4' $ bundle exec ruby -ropenssl -e 'p OpenSSL::VERSION' /path/to/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/runtime.rb:317:in `check_for_activated_spec!': You have already activated openssl 2.0.5, but your Gemfile requires openssl 2.0.4. Prepending `bundle exec` to your command may solve this. (Gem::LoadError)
require :into https://bugs.ruby-lang.org/issues/10320 require 'libfile', into: :Lib Matz has some concerns about ruby internal. • Conflicts versions of a shared library like libyaml-., libssl-, libffi, etc... • Order of LOADED_FEATURE So, This feature is difficult to implement on current ruby specification now.
What are requirements of Rubygems? base64 openssl thread benchmark optparse time cgi pathname timeout digest pp tmpdir English rbconfig tsort erb resolv uri fileutils set webrick find shellwords Win32API io/console socket zlib monitor stringio net/http strscan net/https tempfile
Does Gemification make happy Rubyists? • It’s heavy to maintain like github triage, gem release, resolve dependency. • I should provide a benefit of gemification for Ruby core team.
Summary of Today’s talk • I described about a standard libraries of the Ruby language. • I introduced specification of default gems/bundled gems of Ruby. • I shared current status of Rubygems and Gemification projects. • I shared my plan towards Ruby 3.0.