== Monday, 8 June 2009 == (times in US Eastern) Attendees: Jesse Keating, Paul Frields, Rex Dieter, Seth Vidal, Bill Nottingham, James Laska, Josh Boyer, Luke Macken, Tom 'spot' Callaway, Steven Parrish, John Linville, Matt Domsch (by phone), Bruno Wolff (by phone - mostly observing), Will Woods 0900 - conference convenes, general meet and greet, a little email triage 0910 - Jesse Keating introduces Design Thinking process and the schedule for upcoming days - day1 - define, research, ideate, prototype - day2 - choose a solution ... implementation - day3 - wiki page proposal for larger audience *** THINGS WE'RE DOING WRONG *** - frozen rawhide 11111111111 - GNOME bias 11 - legacy SCM (version control) 111111 - takes too long to compose 111111 - alpha is a lie 111 - beta is pretty much a lie 11 - feature freeze is a lie 1111111 - the cake is a lie 11l - freezes are more of a suggestion (slushy)1 - freeze breaks are hard for the maintainer 11 - Fedora Infrastructure freezes with release freezes (slows things down) - buildroot overrides 111111 - override leaks that corrupt builds - what is rawhide (when does the mirrorlist URL change from rawhide->next_release)? 11111 - SLIPS! 11 - Package Scaling issues... too many packages11 - Every update everywhere 11111 - multilib (?) 1 - signing 11111 - Not signing repos (bz 998) 11 - kopers (we need them) 11111 - daily rawhide, but not daily livecds 1111 - don't close deps on repos... anywhere 11111111 - AUTO QA 1111111111 - fails are not stopped111 - broken update paths 1 - Auto Legal 1 - RC Process (cola), how to notify new images, how to distribute to testers 111 - Rawhide not installable (period) 1111111111 - No last known good rawhide 11111111 - Review process too manual 1 - Broken permissions on release trees1 - Bad package creep 111 - What gets reviewed is not what gets checked in - No push mirroring1 - Chain build 111 - new repo long 1111 - delta'd repodata needed 111 - key people eaten by raptors?111111111 - disk space issues? 1 - Every package is equal (no core)111111111 - Secondary Archs - PPC - PHX2 - Mass Rebuilds - Not in order - Take too long - freeze the system1 - Staging large changes 111 - No auto rebuild for soname bumps 111111 - updates testing (FAIL?) 11111 - No feedback from koji1 - Task Delegation 11 - Testing LiveCDs 11 - Mirror update delay, how to tell if your mirror is up-to-date - Policy for kicking FTBFS packages out 11 - Red Tape 11 - pre/post scripts of RPM are slow 111 - RHEL driven features - Finished test/final release to public availability time too long - Not enough milestones - Can't get stuff to mirrors effectively 1 - Conflicts 11 - Packages 1 - Files 1 - Split CDs - LiveCD vs LiveUSB - Too many Spins 1 - Releng does spins, should SIG's own spin creation? 1 - Spin Popularity, download/usage metrics 11 - Torrent Layout vs Mirror Layout 11 - Inability to QA the full release tree (unclear? does this mean the exploded tree?) 1 - No test result management system11 - RC Test process11 - Spin Feedback on Spin issues? 1 - SIGs are fail 11 - Lack of Enforcement 1111111 - Unable to give feedback 1 - Bugzilla - Too complex for average user1111 - Mirroring takes too long because of how the materials are organized 1 - Few bit-torrent seeders on release-day 11 - comps 1 - What is Fedora? 111 - Where is Fedora going?111 - How to prioritize all these tasks? 11 - OS Release vs repos of packages (Old Core vs Extras problem) 1111 - Inconsistent web client support vs. CLI client support 11 - BZ vs. Trac 1 - Hard to maintain stacks of packages 111 - No limit on features (importance, priority for release)1 ... at this point, we have a local demo of Fedora Community The group cast votes on which problems ought to be the biggest priority. Results in separate "fad-brainstorm-priorities-20090608.txt" document. Cutoff was 4 votes. Items with priority < 4 votes are not shown. First list is raw, second list is sorted by priority. --------------- Frozen Rawhide -- 12 * AUTO QA -- 10 Rawhide not installable -- 11 Don't close deps on any repos -- 8 * Key people eaten by raptors -- 8 Every package is equal -- 10 No last known good Rawhide -- 8 Lack of enforcement -- 7 * Legacy SCM -- 6 Takes too long to compose -- 6 Buildroot overrides -- 6 Feature freeze is a lie -- 5 Every update everywhere -- 5 Signing -- 6 KoPer -- 5 No auto rebuild for soname bumps -- 6 Updates testing (FAIL?) -- 5 What is Rawhide? mirrorlist URL change -- 4 Need daily LiveCDs -- 4 * BZ too complex for average user -- 4 * = After lunch, these problems identified that are mostly outside scope of FAD Devel RDU 2009. Might merit their own FADs. Brief discussion of each ensued. -- LUNCH BREAK -- = Outside of scope for FAD = Removed a few topics as completely outside scope of this FAD. These topics might merit one or more FADs on their own. == AutoQA == (10 votes) - better for a QA-focused FAD. The idea is to create a mechanism to automatically gather QA data (run sanity tests / deps checking / etc), which can be be used for policy later. == Raptors == (8 votes) - Paul requested teams begin to document procedures in the event someone wins the lottery (http://marilyn.frields.org:8080/~paul/wordpress/?p=2500) - possibly get some docs people to 'interview' key people about their procedures to help with wiki-fication. == Legacy SCM == (6 votes) * Spot suggested caution with doing this later, as we may continue to build up infrastructure around the current CVS solution * Be aware of processes that might point in favor of one SCM, but make sure we are not limiting our conversation by thinking only in terms of CVS. == Bugzilla == (4 votes) * Too hard to use * Too slow to get data * Too tied to RHEL * Triage teams need to work together to aid in bug resolution ** Teams with existing triagers and = In-scope item discussion = In reverse order of # of votes. == Daily Live CDs == (4 votes) - produce live images from rawhide daily (or other repeatable time period) - generate data from these images -- size changes -- package changes -- compose failres -- ks changes Failure Point: - THEY DON'T EXIST - risks tests days when Live images are the delivery mechanism for Test days - too big to mirror - too many to compose/monitor - who is responsible for building/testing - nowhere to report issues with images == What is Rawhide == (4 votes) - is it frozen, or open? - is it for the pending release, or the next release? - previous FUDCon discussion around "what is rawhide" - https://fedoraproject.org/wiki/JohnPoelstra/ImproveRawhideF10 Failure Points: - user's don't know what they're consuming - changes out from under users ACTION - merge into later 'Frozen rawhide' topic == Updates-testing == (5 votes) - pending updates to a released Fedora - insulation between stable users and potentially bad updates - published crowdsource QA Failure Points: - people don't use it - those that use it don't know they're using it - those that know they're using it don't report about it - those that do report about it only report the failures (all the news is bad) - those that only report the failures don't enter the failures into bugzilla - there's no notification that the things you have installed from updates-testing went to stable - lack of vision (who's using it and why?) - inconsistent usage by packagers - unknown delay from bodhi request to push - maintainer drive instead of user driven - no dedicated testing - Updates testing is a non-granular firehose == No automatic rebuilds == (6 votes) - react to a scenario to automatically bump and rebuild packages - sonames - deps - ??? - profit Failure Points: - we don't have this - makes stack development harder - leads to broken deps - more work for maintainers - makes kmods and third party repos harder to manage ACTION: out of scope for this meeting == KoPer (koji personal repos) == (5 votes) Ability to build changes of packages in "personal" buildroots without leaking into normal build roots. - chaining scratch builds - better way to do the new maven rebase that's going on for F12 - stack of package reviews. Failure Points: - we don't have this *easily* without releng doing all the work - we don't have this without checking into SCM - not easy to publish the repos - ZOMG too many newRepo tasks CRUSH KOJI - Not possible with packages not in koji Concerns: - Losing to OBS - Private repo proliferation reducing usability and maintainability - Mirroring - Licensing and encumbrance concerns (need AutoLegal) - Lack of developer time - Fear of koji code ACTION: Need to fix newRepo to reduce fail == Signing == (6 votes) - GPG signatures on packages - GPG signatures on repodata - GPG signatures on ISO checksums - Leave gpgcheck=1 on All to provide a verification of the source of the package. - It came from Fedora (koji) and has not been tampered with Failure points: - repodata actually isn't signed - not automated - no distinction on where content in package came from - not in rawhide - only one signature per package - key revoke impossible - key compromise sucks - duplicate data on koji store - time suck at push time - odd churn at resign time (repodata, yum, mirrors, etc.) - we don't trust our own key - secondary arch keys == Every update pushed everywhere == (5 votes) - Making many updates to our stable release - Fedora is about the newest stuff - Fix bugs without waiting to next Fedora release - Keep spec in sync across releases - Want updates on the release they are using - Stay with upstream; don't backport fixes - Feeling useful Failure points: - Huge amount of data to download - Changes product from 'release' to 'release + updates' - Adds instability during freeze - Broken update paths during freezes - More work for maintainers for dependency-based rebuilds - Inconsistency between packager policies and update rate - Documentation gets invalidated - Unclear point of releases - Buildroot pollution - Maintainer driven, not user driven - FEver notifications - bad examples to other packagers - too frequent updates of a single package - not well communicated guidelines - too much to test (by humans) - new packages everywhere RELATES TO: Every package is equal == Alpha is a lie == (5 votes) Success Points: - Provide an early, known good snapshot (in media format) for people to install as a "jump on" point to rawhide (newer than previous GA) - "Some" testing of some features - taking temperature of rawhide - wider audience of rawhide - testing of media based composes and installs Failure points: - Was defined before the feature process, so does not integrate well with it - Features are not fully defined - No features to usefully test - Test results are meaningless with respect to beta/RC/final - Previously, test* releases came before the feature process - doesn't match industry definition of alpha - alpha milestone-making cuts into development time - actually a pre-alpha release - was test 1 and had no meaning other than time GENERAL CONSENSUS: Our current alpha has not been giving us enough value for its baggage = END OF DAY 1 =